Php Transpiler (per to x)

I need to make a transpiler (want is probably a better word) to convert one language (one I create) to another language (a scripting language for a game (.per)).

Here's a simple example of the actual code -

Code:

---

(defrule
        (wood-amount > 100)
        (unit-type-count-total militia-line < 10)
        (can-train militia-line)
=>
        (train militia-line)
)

---

But in my one, it would look like this -

Code:

---

if(
        wood.amount > 100;
        militia-line.count() < 10;
        can(train(militia-line));
){
        train(militia-line);
}

---

How would you go about doing this?
P.s. I know this would be better in a different language, but it's more of a proof of concept...

you'll have to break each language down into its component syntax and logical constructs, and map out directives/conditionals/variables/etc. in one language to their counterparts in the other. Depending on how similar the languages are/ how directly they correlate, this might be difficult.*

*this is my entry in a competition for "understatement of the year."

What's your motivation, here?

Do you need to just convert things in that format? You could use string functions and/or basic regex.

Or do you need something that can actually handle the whole language? You'd need a full parser that also can translate. It's not easy.

Do you really want to take on this project at the moment?

The real problem will come up when there is no linear-order similarity between the languages. If you're only using code for which you can relatively easily substitute chunks in the same sequence, that's just a complex version of a find and replace situation. But if you actually need to go into some sort of higher order logic to figure out which chunks should go where, that'll become much harder.



I've never tried to do this for programming languages, but I've looked into it quite a bit for natural languages. In natural languages, this is more or less impossible because there are too many social/intuitive details that just can't be programmed into a computer, at least not yet. With programming languages you don't have that problem. But from what I've seen for natural languages even if you ignore that, it's incredibly complex. And there's no reason this would be simple, although it would be logically possible.

Thanks for the vote of confidence ;)
Yep, this is what I'm going to take on for the moment.
The languages are very similar, just different formats... though there are some slightly more tricky bits as well...
If you want, I can post bigger samples of both languages?

my vote of confidence was sincere - most people, I wouldn't even consider making a serious response. At most, I'd say "just choose one or the other - to make a translator, you have to be very capable with both languages anyway."

just from looking at your first example, I can see a few spots where you might run into difficulty. for example, how will the parser know that wood-amount should be translated to wood.amount, while militia-line should remain militia-line? You're looking at an actual parser, not search-and-replace.

Are these object-oriented languages (like js)?

1) Create a complete list of correspondences.
2) Create a translation system that switches them while preserving all details.
Easier said than done, but go for it if you'd like :)

One of the main challenges I can see, is getting all the syntax right. After you've actually transpired the language, you could utilise Regex to check if the syntax is correct (first you'll need to figure out what is the correct syntax for particular lines) and if the syntax is incorrect, go through and debug it.

Something will be lost in translation if you are guessing at the end what might be wrong.
But I agree in principle-- there's a huge problem here in that if you need it to work perfectly, the hardest thing is the most important thing-- having it work for most syntax is going to be very limiting and buggy.

If you're serious about this, you'll need to think about the big picture quite a bit before doing it in detail. You might actually find some of the writing on NLP (Natural Language Processing) to be relevant because translation is talked about a lot. It's impossible (at least at the moment) for natural languages, but those same ideas work well for programming languages because the syntax is more controlled and less irregular.

One option to consider is creating a metalanguage that will be easy to translate out of and into, rather than attempting to directly map the two to each other, and that might help you in the end.

If it's not perfect, it won't work (whitespace isn't a problem though).

@Traq: it was sarcasm; *this is my entry in a competition for "understatement of the year." isn't a vote of confidence :p

As for your question Daniel, the difference between militia-line and wood-amount is that militia-line is one object while amount is a sub-object of wood... I'm not sure how to make it detect that though...

Most of the actual data is said the same way across the languages. It's just the bit that does stuff to them is different (in most cases).
It looks like line per line, I'll have to work out the purpose of that line, then pick the appropriate conversion method to convert it.

The only thing I should mention, is that there are some bits that are completly different.
The language per doesn't have any functions() but mine does....

Quote:

---

@Traq: it was sarcasm; *this is my entry in a competition for "understatement of the year." isn't a vote of confidence

---

I fully agree with traq here, in a completely literal sense. This is hard. And perhaps we should underline it and put it in bold: hard. This isn't hard for you personally-- it's hard for anyone to do it. It's logically possible. But it involves not only writing a full compiler but also then being able to make it work with another language. I just can't, honestly, see you actually doing this. I'm not sure how complex these languages are (perhaps they're very simple, and they'd need to be very simple, and maybe then it's possible), but this is right between two of my fields of expertise-- programming and languages, and this sounds like a huge project to me.

Quote:

---

If it's not perfect, it won't work (whitespace isn't a problem though).

---

Then my sincere advice is to not fool yourself into thinking that solving the basics with search and replace is the start and that the rest will come from there. You'll have to start with the hard problems and work this out at a theoretical level including the logic behind why the languages are as they are.

Quote:

---

As for your question Daniel, the difference between militia-line and wood-amount is that militia-line is one object while amount is a sub-object of wood... I'm not sure how to make it detect that though...

---

That's a great example (it was traq's originally) of why this is very difficult. It's the small things that matter. 95% correct isn't correct at all, so it's that hard 5% that you need to worry about, not the easy 95%.
And this is an example of a fundamental issue in language processing: computers don't have common sense. There's no way for it to just "know" that one thing is one type and another thing is another type. You can guess, sure, but the computer can't. There must be a completely formalized (eg, algorithmic) reason for it, or the computer will fail.


There's a problem of infinity here that is a little hard to explain. These languages are recursive (right?) and they can in theory create new and complex phrases. Therefore your system must be able to handle lots of variation. That in itself is hard. But it's also the fact that, for example, symbols can be layered in several ways and you'll need to translate that. You have two tools available:
1) Memorization (eg, telling the computer that x=y)
2) Rules (eg, telling the computer that if there is an X, then if there is a Y, then to do.....)

The one major advantage of this over NLP is that you don't actually have to deal with thousands of words necessarily and lots of variation-- programming languages do have finite lists of parts, usually short lists (in a relative sense) and you could go through and list everything you need. To quote Pānini (Sanskrit grammarian a few millennia ago), 'there are no exceptions, just more specific rules'. This is totally possible, just a lot of work.


Anyway, if you do decide to do this, then I wish you luck, but I can't be much more help than that-- I know where I'd look for answers-- in theories, and wait a while to actually code anything. I don't know if you want to get that deep into it. But I can almost certainly guarantee that you will fail if you don't start at the deeper levels rather than just starting with the easy parts and hoping the rest starts to work out.

It's sort of like building a house-- you can't start with just making the outside look pretty and sort of like a house looks-- you'll never end up at the point where you have the solid foundation with all of the crucial working parts in it.



-----
There is an alternative possibility: maybe this is easy. It's certainly possible that there could be two languages that are so similar that all it takes is a few search and replace operations and you'd be done. If that's the case, sorry for wasting the time getting sidetracked on other cases. But unless these languages are specifically similar in some way, then I expect that is not the case.

Quote:

---

Originally Posted by keyboard1333

*this is my entry in a competition for "understatement of the year." isn't a vote of confidence :p

---

That particular line wasn't my "vote of confidence," nor a vote of "no confidence" - just asserting that it's a very difficult undertaking, almost certainly many times more difficult than you might imagine. Not because of any shortcomings in your imagination, but because you're trying to convince yourself that you can do it. I'm not saying you can't, but that fact does distort your judgement at least a little bit.

Personally, I'd love to see this. It'd be quite interesting, and I'd enjoy helping out where I could. Daniel's right, though; you'd need to do a lot of tedious, complex theoretical work before you started writing any code.

Can you distill your code examples into a basic syntax? e.g.,

PHP Code:

---

$message = $logged_in? 'you are logged in': 'please log in'; 


---

translates to

Code:

---

{handle}{assignment_operator}{condition}?{value if condition is true}:{value if condition is false};

---

and that's not even a particularly complete representation.

I know how hard this is (trust me... the whole phrase "biting off more than you can chew" was based on me [but that's a story for another time]).
The language that I'm converting to is actually incredibly simple.
There is no math, functions or things like that.
There is simply "If these values are true - Do this" (and some minor stuff with declaring constants).
I don't think it's as hard as you're envisioning, but maybe not as simple as I'd imagined...
There are some simple search and replace things (if line starts with // replace // with ; - things like that) but there are also a few tricky things...
Also, it's a very limited language (it does one particular thing and that thing only... no extensions or anything like that).

Basically, it will just be very advanced search and replace script... Not entirely, but mainly....

Quote:

---

Also, it's a very limited language (it does one particular thing and that thing only... no extensions or anything like that).

---

This is very very helpful for what you're trying to do.

Quote:

---

There are some simple search and replace things (if line starts with // replace // with ; - things like that) but there are also a few tricky things...

---

But even your simple example isn't actually that simple. Comments work like that, unless they're inside strings. And what does it mean to be inside a string? You'll start to run into scope issues, and simple search and replace will fail, and regex may fail as well or get very complex.

One important question is whether you rely on well-written code, or any kind of valid syntax.

Quote:

---

Basically, it will just be very advanced search and replace script... Not entirely, but mainly....

---

It's the subset of harder things that will give you trouble. The rest is irrelevant and easy.

Actually, one sucky thing I just though of...
Since this outputs files, I don't particually want them to have to read those files.
And since the engine that runs them only runs files with particular extensions, it would just output an error referencing the outputed script...
E.g. If the user makes a mistake, it would make a mistake in the output.
Therefor, I also have to debug the input....

Yes, that's entirely true, as well as the possibility of oddly formatted (but potentially valid) syntax.
You need to fully parse the code, including checking for errors, then figure out how to translate it.

yes, as daniel mentioned, you will have to format the code yourself to specific standards, as, for example, the user might input all the code on only one line.
so this is what appears to be what the script needs to do.

1) user input
2) format user input
3) check for errors
4) translate
5) check syntax of output
6) re-format it
7) output it

Quote:

---

4) translate
5) check syntax of output
6) re-format it

---

These could be one step if it's written well (eg, logically correct for the entire formal system of the language)

Quote:

---

2) format user input

---

This could also be eliminated if you write a good enough parser, eg, a complete compiler. However, perhaps something like HTML Tidy would be more helpful to get you started. I'm not sure on that one.

.per tidy maybe? One thought I had, was to actually strip some of the whitespace, making it quick to parse. However, that won't work, if per's don't have anything to end a line with (ie, semi-colon), then it just won't parse at all. I'm not familiar with the language, so we'll have to wait for a reply from keyboard.

I'm fairly sure you could just put all the code on one line (each bit of code is wrapped in () ) just like javascript....
I'll go try it to find out....
Might take me a while

Quote:

---

One thought I had, was to actually strip some of the whitespace, making it quick to parse. However, that won't work, if per's don't have anything to end a line with (ie, semi-colon), then it just won't parse at all. I'm not familiar with the language, so we'll have to wait for a reply from keyboard.

---

That's not just an option; it's logically necessary. However, again there's a detail: this must be done only in the right context-- outside of strings, and potentially outside of comments as well. And maybe I'm not thinking of something else. The extra whitespace will be eventually stripped, but not as the first step.

By the way, the idea of tidying up the code should be explained a bit-- it's to make it machine readable, not human readable. So it won't look nice and pretty with the tabs we like; it'll be simple for a machine to understand. In fact, compressing all of it to a single line might be the best way to go in terms of writing a parser-- line breaks are completely irrelevant in most situations in most languages (except, of course, strings, and then there's the issue of whether it has the same 'flexibility' (in my mind bug) of Javascript where a linebreak can be effectively the same as a semicolon).

I've already got it so it strips tabs and other white space.
In retro spect, you can't put it all on one line because the comments wouldn't comment out the rest of the code

does the language you are using have multiline comments? As you could use those, as they have a specific end to the comment:

Code:

---

blahblahblahblahblah/*comment*/blahblahblahblahblah

---

Nope, only "untill end of line" comments.

strip comments?

Huh?
What's that mean???
(Do you mean just strip the comments out because no-one should be reading the src anyway?)

Edit -
Woot: I'm an Elite Coder.... ish

strip the comments, strip the whitespace, parse it. (you were saying that you couldn't strip the whitespace because of the comments)

If it relies on newlines, you'll need to determine what sort of newlines it recognizes (\r, \r\n, etc.) and exclude those from your definition of "white space." You may also need to standardize them.

This is why you'd parse out the comments (and work out string boundaries) before anything else. You could always put them back in later. But a real parser doesn't parse comments, so it can represent everything on one line. You're translating, so you'd like to keep the comments, which is fine, but you might need to do a temporary replacement with a variable in your metalanguage (eg, "$comment1", or whatever kind of metalanguage representation you'd like).

Actually, I don't need to keep the comments as the whole point is that you don't need to look at the output....

Right now, It strips the whitespace from the beginning and end of each line, and next I'll have it strip comments and escape all the text within the chat functions (so that it doesn't try to replace them with anything).
I can actually do practically everything I need with some code you gave me a while ago traq -

Code:

---

$find = array(
    "1" => "#(chat_all\(\")(.*)(\"\)\;)#ui",
        "2" => "#(break\(\)\;)#ui",
        "3" => "#(if()#ui",
);
$replace = array(
    "1" => "(chat-to-all \"$2\")",
        "2" => "(disable-self)",
        "3" => "(defrule",
);
echo '<pre>' . preg_replace($find,$replace,$handle) . '</pre>';

---

Stuff like that...

Quote:

---

practically everything I need

---

That's exactly what I warned you not to do-- it's misleading and you will eventually fail when you find that the 5% of hard things aren't compatible with that model of translation. You can do whatever you'd like, but I can't help you unless you start with the theory of translation and working on this at a more abstract level.

Unless you can do everything you need (not practically everything) that way, it's not going to be helpful (at this point) to do search and replace. The replacing will eventually be used, but not the searching, not in that way.

Oops...
The only thing is; I can't think of anything that can't be done in that way...
And the only way to find those things is to try them all out.... correct?

What do you mean look at it on a more abstract level?

Quote:

---

The only thing is; I can't think of anything that can't be done in that way...

---

I don't know the full syntax of these languages. If there really is nothing that can't be done by a simple "flat" search and replace, then go for it. But if there is anything that requires re-ordering of parts of the syntax, you'll get into serious trouble trying to do it that way.

Quote:

---

And the only way to find those things is to try them all out.... correct?

---

In some sense. But understanding the theory behind it will help you predict the problems.

Quote:

---

What do you mean look at it on a more abstract level?

---

My other life is doing linguistics and I've seen quite a bit on this topic. A friend of mine was trying to write a Swahili-English translator and had some promising results, just using formulas. But the problem is that you will eventually run into problems for certain things that don't exactly fit those patterns. And that's the hard part.
There are a couple ways to approach this; what I'd do is parse it into some sort of representational language so that you can then easily translate between them. But that maybe isn't necessary.

I suggest spending a day reading about the topic; here's one place to start:
http://en.wikipedia.org/wiki/Translator_%28computing%29
Another is to look into "Machine Translation", which will be about natural languages-- it'll go into certain topics you do not need to worry about, but overall a lot of the ideas will be helpful.

There are some bits that require complete re-ordering...
E.g. I can add in functions to my language by using "goals" (if value = something, goal = something. Then if goal = something, do something), but that means writing stuff to the top file and all kinds of stuff...
I'm fairly sure I can do this just with things like preg_replace, but the code would be so complex it wouldn't be funny....

Ahh... so your talking about intermittent languages?
I've actually done a fair bit of research on compilers all ready (and things related to them) but I'll go have a look at what your suggesting...

My only question is: how would using an intermittent language actually stop the need for search and replace?
I mean, how is doing that actually any different to the preg_replace and stuff like that?

Well, having an intermediate language would allow you 2 things:
1) If you ever want to expand this beyond just X-Y translation you'd be able to do it more easily (adding also, let's say, PHP translation, or JS translation).
2) The process of writing the new code will be much easier than parsing the old code, no real question about that. And that will mean that you don't need to worry about oddities of the target language while parsing the oddities of the source language. Your intermediate language would need to have all of the functionality of each but be organized in a very clean way such that you know exactly how to translate into another languages-- it wouldn't even need to be a "language" in a strict sense, but more like a set of instructions to your program to write code into a different language.

Ahh... ok.
Just found this.... does that look like the kind of thing to do?
Actually, there's no point in being able to convert to different languages: the language that I'm converting to is the only one that can do what it does.
Sure you could use a similar syntax, but the actual values and stuff would never be used in a different language...

That looks completely reasonable to me, and it looks similar to what I've seen in NLP, in some fundamental ways, not in the details of course.

I've got an actual coding question -
I'm trying to make two arrays - the value in array1[1] should be the data related to array2[1] (and so on).

I've got this -

PHP Code:

---

<?php
$handle = file_get_contents('Test/1.per');
echo '<pre>' . $handle . '</pre><br /><br />';
$char_type = array(
    " " => "SPACE",
    "bar" => "foo",
);
$export;
$count1 = strlen($handle);
$count2 = count($char_type);
for($i=0;$i<$count1;$i++) {
    $r = substr($handle, $i, 1); 
    for($u=0;$u<$count2;$u++) {
        if($r = $char_type[$u]) {
            $compare[i] = $char_type[$u];
            //break;
        }
    }
    $export[i] = $r;
    echo $r;
    //break;
}
echo $count2;
for($z=0;$z<$count1;$z++) {
    echo $export[z] . '<br />';
    echo $compare[z] . '<br /><br />';
}
?> 
<!DOCTYPE html>
<html>
<head>
</head>
<body>
</body>
</html>

---

And the file 1.per -

Code:

---

if(
        (or
                game.time > 1;
                game.time > 100;
        )
        count(militia-line) < 10
){
        chat_all("Hello World");
        break();
}

---

The php code isn't writing the two arrays (but it's writing the contents of the file).
Any help?
Also, is there a better way to do this?

Shouldn't there be $ signs with the arrays and loops?

Code:

---

<?php
$handle = file_get_contents('Test/1.per');
echo '<pre>' . $handle . '</pre><br /><br />';
$char_type = array(
    " " => "SPACE",
    "bar" => "foo",
);
$export;
$count1 = strlen($handle);
$count2 = count($char_type);
for($i=0;$i<$count1;$i++) {
    $r = substr($handle, $i, 1);
    for($u=0;$u<$count2;$u++) {
        if($r = $char_type[$u]) {
            $compare[$i] = $char_type[$u];
            //break;
        }
    }
    $export[$i] = $r;
    echo $r;
    //break;
}
echo $count2;
for($z=0;$z<$count1;$z++) {
  echo $export[$z] . '<br />';
    echo $compare[$z] . '<br /><br />';
}
?>
<!DOCTYPE html>
<html>
<head>
</head>
<body>
</body>
</html>

---

PHP Code:

---

<?php
$handle = file_get_contents('Test/1.per');
echo '<pre>' . $handle . '</pre><br /><br />';
$char_type = array(
    " " => "SPACE",
    "(" => "LEFT BRACKET",
);
$export;
$count1 = strlen($handle);
$count2 = count($char_type);
for($i=0;$i<$count1;$i++) {
    $r = substr($handle, $i, 1); 
    for($u=0;$u<$count2;$u++) {
        if($r = $char_type[$u]) {
            $compare[$i] = $char_type[$u];
            //break;
        }
    }
    $export[$i] = $r;
    echo $r;
    //break;
}

for($z=0;$z<$count1;$z++) {
    echo $export[$z] . '<br />';
    echo $compare[$z] . '<br /><br />';
}
?> 
<!DOCTYPE html>
<html>
<head>
</head>
<body>
</body>
</html>

---

Fixed what you pointed out but it still doesn't work...

One thing I can see, or can't see rather, is where the variable $compare is created. You actually need a variable before you should write to it. Shouldn't you just create compare and export as empty arrays rather than one empty variable, and one not-a-variable?

Code:

---

<?php
$handle = file_get_contents('Test/1.per');
echo '<pre>' . $handle . '</pre><br /><br />';
$char_type = array(
    " " => "SPACE",
    "(" => "LEFT BRACKET",
);
$export = array();
$compare = array();
$count1 = strlen($handle);
$count2 = count($char_type);
for($i=0;$i<$count1;$i++) {
    $r = substr($handle, $i, 1);
    for($u=0;$u<$count2;$u++) {
        if($r = $char_type[$u]) {
            $compare[$i] = $char_type[$u];
            //break;
        }
    }
    $export[$i] = $r;
    echo $r;
    //break;
}

for($z=0;$z<$count1;$z++) {
    echo $export[$z] . '<br />';
    echo $compare[$z] . '<br /><br />';
}
?>
<!DOCTYPE html>
<html>
<head>
</head>
<body>
</body>
</html>

---

give that a try.