What is Extended Code - Part 2
Which One Should I Use, Part XVIII
Phil Kingery | Advanced Control Technologies, Inc
In the political satire, "Gulliver's Travels" by Jonathan Swift, there were the 'Big Endians', who broke their eggs at the large end ("the primitive way"). They rebelled against the Lilliputian King who required his subjects (the 'Little Endians') to break their eggs at the small end. What, you may ask, does this have to do with X-10 "Preset Dim" commands? Well, big-endian and little-endian are also terms used to describe the order in which a sequence of bits and bytes are sent, stored and displayed.
If you already have a pretty good working knowledge of binary and hexadecimal, and you understand how to convert from one to the other, then this tech tip may be for you. However, if you have no idea what I am talking about, I suggest you go back and read #14 and #17 in this series. Even so, I still must warn you that you are sailing into dangerous waters. We are still working up to getting in really deep into "Extended Code", but there is still some binary information with which we have to contend. This article is specifically intended to explain which commands changed when X10 decided to update the protocol. This article will not go into how to use extended code. This is a "transition" article.
First, let's go over some terms:
X-10= When I use "X-10" (with the hyphen) in a sentence, I mean the protocol, not the company.
X10 = When I use X10 (no hyphen) in a sentence, I mean the company.
Basic Code = In WOSIU#15, I said that: "Ö the X-10 protocol is not a "standard"Ö" However, when discussing "Extended Code" as it relates to the older command set, we usually use the term "Standard Code" (as in "Standard Code" versus "Extended Code"). So, in order to reduce the confusion, I have decided to use the phrase "Basic Code", (as in "Basic Code" versus "Extended Code").
Extended Code = This is not a simple definition. With the publication of the extended code protocol at ftp://ftp.x10.com/pub/manuals/xtc798.doc, X10 said that there would now be three different "extended code" data frames.
Extended Code 1 - For Data/Control(bit pattern 0111, which is the foundation of this article)
Extended Code 2 - For Meter Read & DSM (bit pattern 1100)
Extended Code 3 - For Security Messages (bit pattern 1010)
As far as I know, only "Extended Code 1" is defined. Therefore, for the purposes of this article, when I use the term "Extended Code", or the short "Ext'd Code", I mean it to mean "Extended Code 1".
First: The Demise Of The Old Preset Dim Commands.
All Units Off, All Lts On, On, Off, Dim, Bright, All Lights Off, Ext Code, Hail Request, Hail Acknowledge, Preset Dim (1), Preset Dim (2), Extended Data, Status On, Status Off,and Status Request.
Then, X10 decided to amend the list of commands. There were four commands that changed. Two of those were of particular interest since they were commands that X10 themselves never used. That is not to say that those commands were never used at all, it's just that other companies, other than X10, used them. Those commands were "Preset Dim (1)" and "Preset Dim (2)". Since X10 came up with the protocol, they decided that they were going to abandon those two commands. Since the X-10 protocol is unregulated (it is published but all the patents have expired and there is no governing body that enforces compliance) these older codes continue to be used in many products, contrary to X10's intent. Before we can understand the new extended code way of doing it, we need to be very clear on how the old way was supposed to work.
A quick note on some "Uncle Phil" notation here: You will notice that when writing address codes, I don't write "A1". I try always to use 3 characters, as in "A01", so that it makes sense when using devices which require consistency. I also try to use commas to denote contiguous data, such as "A01,A01", which is meant to show that there are no gaps between the two data frames. I use a dash, such as "A01,A01-AOn,AOn", to show that there are at least 6 blank zero crossings between data frames.
We all know that when sending an "On" command to a receiver addressed to "A01", we need to transmit: "A01,A01-AOn,AOn". The X-10 protocol is based upon the idea that every command is linked to a letter code and that letter code is the same as the receiver for which it is intended.
The only exceptions to this rule were the old "Preset Dim" commands. Before we get into how those commands were 'supposed' to work, allow me a little side issue. The very term "preset" sounds more like a scene, than what it really was, which was a "direct dim" command. In other words, the idea was for an X-10 transmitter (controller, et al.) to be able to send an address and a command which caused the dimmer receiver to go 'directly' to one of 32 different levels, no matter if it was above that, below that, on or off.
So, from now on, when I say "Preset Dim" in this document, I really mean "Direct Dim" in my own head. Okay?
There is another problem of terminology. In X-10's own document at: http://www.x10.com/support/technology1.htm it only says "Pre-Set Dim", but never says "Pre-Set-1" or "Pre-Set Dim 2" or anything like that. All it says is that the D8 bit is the most significant bit (etc.). The document at ftp://ftp.x10.com/pub/manuals/cm11a_protocol.txt uses the terms:
Pre-set Dim (1) 1010-and- Pre-set Dim (2) 1011
However, we at ACT (internally) use the terms, "PR0 = Preset Dim 0" and "PR1= Preset Dim 1". This makes a lot of sense since that more closely relates to the binary representation of bit D8. (It also fits well when using devices which transmit the commands (like the AT004 or the TI103-RS232). However, having said that, I believe that would be darned confusing to most people reading this, so for purposes of this article I will use the terms "Preset Dim (1)" and "Preset Dim (2)" (or abbreviations thereof).
Everybody keeping up so far? Raise your hand if you don't understand. Okay, let's move on.
Since X-10 never implemented the use of these older "Preset Dim" commands, they are currently interpreted in whichever way anyone wants to use it. So as to not 'exclude' anyone else's method of interpretation, our software manager decided to allow our computer interface (the TI103-RS232) to send the commands in the "lowest common denominator", instead of trying to send dim levels or percentage dimmed values. In other words, he so wisely decided to have the TI103 and AT004 send the letter code and then either one of the "Preset Dim" commands as the user needs, and then let whomever is receiving it, to interpret it as they wish.
Now, let's go though how it was 'supposed' to work. Quoting from my own, "Which One Should I Use, #15: Two-Way Receivers with Direct Dimming and Scenes" article, I say (edited slightly):
In order to cause the dimmer (set to A01) to go directly to one of 32 different dim levels (0-31), the user could transmit one of these data strings:
A01,A01 - M-Preset-Dim1,M-Preset-Dim1 = 00000 = dim level 0 (0%).
A01,A01 - N-Preset-Dim1,N-Preset-Dim1 = 10000 = dim level 1 (~3%).
A01,A01 - O-Preset-Dim1,O-Preset-Dim1 = 01000 = dim level 2 (~6%).
A01,A01 - L-Preset-Dim2,L-Preset-Dim2 = 10111 = dim level 29 (~94%).
A01,A01 - I-Preset-Dim2,I-Preset-Dim2 = 01111 = dim level 30 (~97%).
A01,A01 - J-Preset-Dim2,J-Preset-Dim2 = 11111 = dim level 31 (100%).
As you can see, the idea was that the transmitter would still send out the address as normal, but after that, the letter code of the command section would no longer match the letter code of the address. The letter code would now designate the "direct dim level".
Since A-P=16 letters, there would be 16 available levels with "Preset Dim (1)" and 16 more levels with "Preset Dim (2)", for a total of 32 levels. However, there appeared to be a problem with the bit order. No one is disputing the lowest letter in the sequence. Since the letter code "M" is 00002, that has to be 0% (when sent with Preset Dim 1). The highest level is also indisputable. The letter code "J" is 11112, so it has got to be the highest value (when sent with Preset Dim 2). But, what letters go in the middle is an interesting story.
In the political satire, Gulliver's Travels by Jonathan Swift, there were the 'Big Endians', who broke their eggs at the large end ("the primitive way"). They rebelled against the Lilliputian King who required his subjects (the 'Little Endians') to break their eggs at the small end.
What, you may ask, does this have to do with X-10 "Preset Dim" commands? Well, big-endian and little-endian are also terms used to describe the order in which a sequence of bits and bytes are sent, stored and displayed. Big-endian is an order in which the "big end" (most significant value in the sequence) is stored first (at the lowest storage address). Little-endian is an order in which the "little end" (least significant value in the sequence) is stored first.
Let me explain this in a little easier way. Let me see how much money I have in my pocket right now. Okay, I have one ten dollar bill, three dollar bills, plus 89 cents in change. You and I (and pretty much everyone else on this planet) use the big-endian method of notation even though we never really thought about it. We list the highest value numbers on the left, and the numbers go down in value each time we move a place to the right. In my pocket, I have $13.89 which means, 1 ten dollar bill, 3 one dollar bills, 8 tenths of a dollar and 9 hundredths of a dollar. The smallest numeral (the "1") has the highest value because of where it is placed (far left). The largest numeral (the "9") has the smallest value because of where it was placed (far right). Perhaps on another planet, they may use the little-endian method and so my counterpart may have the exact same amount of money in his pocket, but he writes it as 98.31$.
Remember that the X-10 protocol came about at the beginning of the digital revolution. So, it is not hard to see that they probably used the little-endian method to send and display the bits. In other words, we think of the letter "N" as being "10002" which we all equate to being 810. Instead (although it really doesn't matter as long as all the X-10 compatible devices are consistent), I think that "10002" had the "1" in the little-endian position, since that was sent first onto the powerline, and therefore, that would mean that it really equates to 110.
The chart that I created for WOSIU#17, lists the letter codes, number codes and commands in the more common "big-endian" method, but in the early days of digital signal transmission it was common to send (and therefore display) the bits in the "little-endian" method. Knowing that, this chart makes a lot more sense as far as the old "preset dim" commands.
Now, the order of the letters makes a lot more sense.
This "big-endian" / "little-endian" thing should have been more obvious to me if I had looked a little closer. In the original bit pattern chart that is published by X10 (at http://www.x10.com/support/technology1.htm), it clearly labels the columns in ascending order from left to right: "H1 H2 H4 H8". However, in later documents, X10 lists the same columns in descending order from left to right: "H8 H4 H2 H1". That would have been alright, if when they swapped around the column values, they had also swapped around the bit pattern, but they didn't.
Therefore, somewhere between the original publication of the X-10 protocol, around 1978, and the publication of the extended code protocol, around 1993, the value of every non-palindrome bit pattern was reversed. And with that sudden realization, I feel extremely proud of myself!
Regardless of the bit pattern order or values, there is another point of discussion. Every command is deliberately preceded by the same letter code as the address to which it is linked. Every command, that is, except the preset dims. This has the potential of really causing problems in a complex X-10 system. Since the X-10 protocol is limited in baud rate by the very sine wave itself (either 60Hz or 50Hz on this planet), the X10 engineers could not afford to include sophisticated checksums, nor cyclic redundancy checks or even a parity bit to increase dependability. They only had two ways to do it. First, they used what is called "Manchester encoding". That meant that every "1" bit was actually sent with its complement, or 1+0. Every "0" bit was sent with its complement, or 0+1. This greatly increased the reliability of the data frame since single-bit errors were quickly identified and the data frame could be discarded.
The second method of increasing reliability is to always include part of the address (in this case, the letter code) in every command. There is no such thing as an "On" command. Instead, it is "AOn", or "BOn" or COn, etc. However, with this "Preset Dim" format, that is no longer the case. The letter code of the address and the letter code of the command are now different (87.5% of the time). The command was no longer directly linked with the address of the device for which it was intended. Instead, the "dim level" is the letter code. This may help to explain why X10 abandoned this method in lieu of the more precise (and more reliable) extended code method. There are more problems with this old "Preset" dimming method (and why we at ACT held off on it for so long), and that is that some "controllers" only use 16 (every other one) of the 32 available levels. There has also been some confusion as to where you start counting; is "2" just a little down from the brightest, or just a little above "Off". Another point of misunderstanding is from the fact that most non-technical users are uncomfortable using 0-31, so instead use 1-32.
I have never been told the exact reason why X10 decided to abandon the old "Preset Dim" commands. It could have been that they realized that the potential for errors was too great. It could have been that the emergence of extended code made the older preset dims unnecessary. It could have been something else. Or, as I suspect, it was a combination of all of these reasons.
Now, having gone through all this trouble to explain the old "Preset Dim" commands (which I prefer to call "Direct Dim" commands), you should understand this: X10 no longer supports them. Other companies (including ACT) have products that do use these commands and regardless of why X10 never used them or why they decided to abandon them, it is a done deal. All of us who continue to use them, do so without the support of X10.
This could conceivably cause a problem in the future. For instance, if some new X10 product which uses "Extended Code 3" is installed into a facility with an ACT coupler-repeater, our repeater would not be able to correctly identify it as an "Extended Code 3" data frame. Instead it would identify the bit pattern as being an old "Preset Dim (1)" command and start repeating the first part of the data frame before the transmitter was finished sending the second part of the data frame. Less disastrous would be for an X10 product capable of sending an extended code 2 data frame was installed with an ACT repeater. If that would happen, our repeater would simply ignore the data frame since we know that we can't do anything with it. So far, we have never heard of any such occurrence of either situation, but one day, it may happen.
Second: Two Other Commands Also Evolve.
The old basic code had listed commands for "Preset Dim 1", "Preset Dim 2", "Extended Code", and "Extended Data". However, after the change, "Preset Dim 1" became "Extended Code 3", "Preset Dim 2", became "Unused", plain old "Extended Code" became "Extended Code 1" and finally "Extended Data" became "Extended Code 2". We have already talked about preset dims to death. Now, we can take a paragraph or two to discuss the other two.
First, let me discuss the old "Extended Data". X10's documentation stated that the extended data command was followed by 8 bit bytes which can represent analog data. It further stated that the first byte can be used to say how many other bytes of data would follow. That means that depending on the situation, the basic data frame (11 cycles) could be followed by one more byte (8 more cycles of the powerline sine wave), or that byte can be used to say how many other bytes would follow. That implies that there could be anywhere from 8 more cycles, all that way up to 2056 more cycles. That would be a very long data frame. Instead of less than a second to send the data, it could be 34 seconds or more to send it and that is just to send it once..
Since ACT has long been known for its coupler-repeaters, having an indeterminate data frame length, makes it impossible to design a practical repeater. If the coupler-repeater doesn't know how long the data frame is going to be, how will it know when to start repeating? Even though "Extended Data" has become "Extended Code 2", the problem remains. We still don't know how long the data frame is (and it could vary), so we have chosen not to try and repeat it.
The old "Extended Code" had the same problem. It was also undefined as to how long it could be. X10 documentation said that "bytes" could follow (but exactly how many, is up to the designer) and that it could "expand beyond the 256 codes presently available". However, no details are given. Fortunately, The old "Extended Code" became the new "Extended Code 1", which does have a specified data frame length (31 cycles) and so repeaters were designed to know when to start repeating.
The structure of the X-10 "duplicate frame" format is inherently easy to "repeat". This repeated frame is not to be confused with the original signal's duplicated frame. Instead, this is intended to describe the ability to design and construct a separate device that increases the range of the commands. This is done by having the coupler-repeater constantly listen for the existence of frames of data. When a frame is received, it will then repeat that frame over the next 11 cycles (as illustrated below).
Since the basic X-10 data frame is of a known length, the action of repeating it does not consume any additional line time. Therefore, a repeater does not compete with other transmitters for line access. This may explain why ACT has occasionally been asked to develop repeaters for other powerline protocols (which we have declined). Most other powerline protocols have a variable length data packet and (as far as is understood) is never sent in a duplicate frame format. It would appear that the only way to successfully repeat those data packets would be to receive the data, store it, then wait to see if there is a response. If not, wait for a clear spot (according to the hierarchy of access parameters) and then re-transmit the data. Should the line have unusually heavy communications traffic, the "repeater" may have to store a large number of data packets before an opportunity would permit the retransmission of the data. At that point, the original frame (and perhaps other frames held in a buffer) may be obsolete.
Having a defined data frame length gives X-10 basic code systems a huge advantage over other protocols. At the time of this article, only "Basic Code" and "Extended Code 1" have a defined frame length (which is 11 cycles and 31 cycles, respectively). The 31 cycle frame length for "Extended Code 1" is described as: "Start code = 4 bits, Housecode = 4 bits, Extended code 1 = 5 bits (01111), Unit code (device code) = 4 bits, Data = 8 bits, Command = 8 bits". "Extended code 2" is variable in length, depending on the type of message. Extended code 3 has been "assigned" for security but doesn't actually exist yet so its format has not yet been defined." Refer to X-10's own XTC797 document for more information.
I think this is a good place to stop. I think I just heard the school bell 'ding' and you are probably in need of a brain break. I realize that a lot of this information is very detailed and much of it is not going to be of much help to a lot of you. Some of you, however, may be struggling with the fine details of trying to develop software for your front-end controllers or new products. You may also be struggling with the decision of having ACT design and manufacture your new products for you. Even if you don't understand all the nuances of this technology, I wanted to let you know that we do.
We will continue this in the next article.
-"Uncle" Phil Kingery. firstname.lastname@example.org
The content & opinions in this article are the author’s and do not necessarily represent the views of HomeToys
This post does not have any comments. Be the first to leave a comment below.
Post A Comment
You must be logged in before you can post a comment. Login now.