ECU Diagnostics - CAN Bus, hunt for answers

[Purplemeanie]

Looks like our messages crossed Mark, I was responding to your previous one.

Now for your last message...

Yes, PiCan2 is SocketCan compatible... seems to work well, once I'd found the one I'd squirrelled away from 2017!. They also do variants with GPS and accelerometer built in, and one that has two CAN interfaces that can be used as a bridge. Let me know if you get to try one as there were a couple of twists to the driver setup for the latest 4.x kernels.

i should probably do a blog post on its setup, but I was hoping we'd crack the code before I did that :-)

[CtrMint]

John, As requested, https://www.dropbox.com/s/ykr8pquykireqik/pcap_as_candump.txt?dl=0

Guys,

I've created a Slack channel.  Might be easier to discuss, exchange files etc.

 

[CtrMint]

And avoid duplicates 

[CtrMint]

Morning All,

I've been giving some of the basics more thought following John's data and I've come to the conclusion we might be making some incorrect assumptions.

I think we've all agreed the ID represented a device communicating on the bus, I certainly did making the analogy with an IP MAC address. I also believe John suggested similarly when describing the outcome of his work yesterday. For example John suggested one of the additional IDs represented the SBD interface etc.

Having reviewed a lot of documentation relating to CAN and many other none 7 applications, I'd like to suggest the ID isn't presentative of a device. It is a message identifier, which describes the intent of the message. The payload contains data associated with the predefined message.

There is a great blog article here which describes how the ID is used for various systems and messages on a Jeep.  I suggest this is the appropriate use of the ID.

http://chadgibbons.com/2013/12/29/hacking-the-jeep-interior-can-bus/

Unlike many mainstream cars which have plenty of CAN enabled devices on the bus we have one, the ECU. Other than engine management it does very little. It's for this reason we aren't seeing many other IDs, those we do see relate to logging. It's a coincidence the number of ID values observed is similar to the devices connected.

John showed yesterday the ECU will spit out the same data as mine until he connected Easymap. That resulted in a 0x(03,04,00,0d,ef,fb,ff,f7) packet being sent. After which the ECU started streaming additional IDs.

As a result, I propose the ECU sends a set of preample/announcement packets with the ID 0x0cbb0001 and various data/values, let's call them key/value pairs (KVs). These have some basic meaning. When received by Easymap, I suspect one or a couple will be recognized by Easymap and cause Easymap to load the appropriate EC2 file. At which point Easymap sends back the packet John flagged. In response the ECU dumps the other ID and KV pairs. The additional ID contains the logging stream.

There is of course a chance MBE hasn't followed the CAN standards, and used the protocol in a manner where the ID represents a hardware or device identifier, but I now find that hard to believe.

This theory needs testing.

I suggest replaying the 0x0cbb0001 frames at Easymap, to see if it will load an EC2 file. Hopefully if it works, we could also further validate the theory by changing the ID but keep the payload the same. A changed ID shouldn't illicit the same response as it would be ignored by Easymap since it wouldn't represent the announcement frames.

Hope that makes sense.

Mark

[Purplemeanie]

So a bit of a diversion today... I was looking at the Easimap protocol again as Mark and I were chatting on Slack and I remembered I had an OBD scanner that I bought when I was building the car. So I decided to connect that back up to the car again and see what it did on the CANBus with my newly set up Raspberry Pi CANBus sniffer.

By the way, some of you old hands will probably know all this already. But hopefully its helpful to some of you.

The scanner is an Autolink AL319. It goes through a setup phase where it tries to decide what protocol the car is talking - Which turned out to be "CAN EXD FAST". Makes sense seeing as the Wireshark captures we've been looking at report exactly that (CAN EXD).

It then goes through a phase of deciding what data the car can report using the OBD protocol.

It's perhaps worth noting here that these ECU's talk at least two protocols. The one Easimap uses and OBD-II. The OBD protocol is quite easy to decode but the Easimap one is proving to be a little more troublesome.

Anyway, that lead me to what OBD is, and to this page:

https://www.csselectronics.com/screen/page/simple-intro-obd2-explained/language/en

and then to the Wikipedia page here:

https://en.wikipedia.org/wiki/OBD-II_PIDs

If you want to dig into OBD then those two pages explain it all. But suffice to say the protocol is a Request/Response protocol. The Scanner sends a request for a response about a PID (see the two links for more info on PIDs).

The upshot of my research was that I could see the car responding to three PID discovery requests as follows:

9A4 ECU Response the OBD PID 0
0xd8368019 -> 0b11011000001101101000000000011001

9A4 ECU Response the OBD PID 20
0x20002001 -> 0b00100000000000000010000000000001

9A4 ECU Response the OBD PID 40
0x40100000 -> 0b01000000000100000000000000000000

And that leads to the following PIDs being supported by the car. Just thought some of you might appreciate this research. It seems that some websites refer to Services (the new term) and some to PID, they are the same thing as far as I can see.

(I've checked my working out and I think I've got it right, but someone might like to check it)

To be clear: This data is not present on the OBD/CAN port by default. You have to have a device connected that can "ask" for the data. i.e. the device has to send a "request" and the car sends the "response".

Service (PID) Supported (Snn)
S01 1 -> Monitor status since DTCs cleared. (Includes malfunction indicator lamp (MIL) status and number of DTCs.)
S02 1 -> Freeze DTC
S03 0 -> 
S04 1 -> Calculated engine load
S05 1 -> Engine coolant temperature
S06 0 -> 
S07 0 -> 
S08 0 -> 
S09 0 -> 
S0A 0 -> 
S0B 1 -> Intake manifold absolute pressure
S0C 1 -> Engine RPM
S0D 0 -> 
S0E 1 -> Timing advance
S0F 1 -> Intake air temperature
S10 0 -> 
S11 1 -> Throttle position
S12 0 -> 
S13 0 -> 
S14 0 -> 
S15 0 -> 
S16 0 -> 
S17 0 -> 
S18 0 -> 
S19 0 -> 
S1A 0 -> 
S1B 0 -> 
S1C 1 -> OBD standards this vehicle conforms to
S1D 1 -> Oxygen sensors present (in 4 banks)
S1E 0 -> 
S1F 0 -> 
S20 1 -> PIDs supported [21 - 40]

S21 0 -> 
S22 0 -> 
S23 1 -> Fuel Rail Gauge Pressure (diesel, or gasoline direct injection)
S24 0 -> 
S25 0 -> 
S26 0 -> 
S27 0 -> 
S28 0 -> 
S29 0 -> 
S2A 0 -> 
S2B 0 -> 
S2C 0 -> 
S2D 0 -> 
S2E 0 -> 
S2F 0 -> 
S30 0 -> 
S31 0 -> 
S32 0 -> 
S33 1 -> Absolute Barometric Pressure
S34 0 -> 
S35 0 -> 
S36 0 -> 
S37 0 -> 
S38 0 -> 
S39 0 -> 
S3A 0 -> 
S3B 0 -> 
S3C 0 -> 
S3D 0 -> 
S3E 0 -> 
S3F 0 -> 
S40 1 -> PIDs supported [41 - 60]

S41 0 ->
S42 1 -> Control module voltage
S43 0 ->
S44 0 ->
S45 0 ->
S46 0 ->
S47 0 ->
S48 0 ->
S49 0 ->
S4A 0 ->
S4B 0 ->
S4C 1 -> Commanded throttle actuator
S4D 0 ->
S4E 0 ->
S4F 0 ->
S50 0 ->
S51 0 ->
S52 0 ->
S53 0 ->
S54 0 ->
S55 0 ->
S56 0 ->
S57 0 ->
S58 0 ->
S59 0 ->
S5A 0 ->
S5B 0 ->
S5C 0 ->
S5D 0 ->
S5E 0 ->
S5F 0 ->
S60 0 ->

John

[Purplemeanie]

Progress Report:

Mark and I haven't given up on this project (yet)!

We've been setting up test systems and figuring out what's what on the OBD port.

It's clear that the MBE 9A4 ECU's use a couple of different protocols on that port. There's an MBE specific protocol that's used by Easimap to talk to the car, and there's OBD-II.

In order to decode the OBD traffic I had been using the Wireshark packet sniffer. Wireshark has recently updated its support for OBD and it can now decode 11bit CAN OBD IDs. However, it struggled (that's software speak for: it didn't) when we tried to get it to decode the 29bit codes used on our MBE ECUs.

So, to get that fixed I've worked this week on getting Wireshark patched for our OBD 29bit protocols and have now submitted a patch to allow Wireshark to correctly dissect our OBD port.

Here's a screen shot of what the OBD protocol now looks like with my patch in Wireshark:

The patch is here:

https://code.wireshark.org/review/#/c/34297/

This is my first Wireshark commit so I'm not quite sure how the process runs for getting it into the regular Wireshark releases, but hopefully it won't take too long.

If anyone's interested, Wireshark can be downloaded from here (obviously doesn't contain my patch yet):

http://wireshark.org

The next step is to try and decode the MBE Easimap protocol. Mark and I have made some good progress on this but we're still a long way from having it cracked.

More to follow over the coming days/weeks...

John

 

[CtrMint]

Probably also worth mentioning the Slack channel again.  John and I are communicating pretty much daily over Slack.  There's an open invitation to anyone wishing to contribute to the development of this project.  Please PM me.

[Purplemeanie]

Another Progress Report:

We (CTRMint, Aerobod and myself) have made some good progress since the last update.

Our work is with the 9A4 MBE ECU and the car's OBD connector with the following working assumptions:

• There are three protocols used by the ECUs
  • A Broadcast Protocol with limited information that outputs about every 100ms. 
  • OBD-II standard protocol that is a request/repsonse protocol based around 29bit CANbus IDs
  • Easimap ISOTP-like protocol that is a request/response protocol using 29bit CANBus IDs and a modified ISOTP (ISO-15765) protocol

Broadcast Protocol

There are a few CANbus messages broadcast by the ECU's using a CAN ID of 0x0cbb0001. These messages are sent as soon as the car is put into ignition position 2. There are probably some useful data entries in these messages but there's not a lot there so we haven't concentrated on that yet.

OBD-II Protocol

The MBE ECU's support the standard OBD-II protocol. You can send the ECU OBD-II data requests and the ECU will respond. We've seen this protocol being used by OBD scanners and it mostly makes sense.

We've been using the Wireshark packet sniffer to decode the OBD-II protocol and have submitted a code patch to Wireshark.org to allow Wireshark to decode the 29bit CAN IDs, modes and PIDs that the MBE ECUs use. This patch has now been accepted and should hopefully appear in the next release of Wireshark.

EasiMap ISOTP-like Protocol

The third protocol is probably the most interesting and where we have spent the most of our time. This protocol is used by Easimap 6 to communicate with the ECUs and has a lot more data variables than either of the other two protocols.

This protocol is also a request/response protocol where a querying device (Easimap with MBE 985, or some other CANbus tool like a Raspberry Pi with a PICAN2 HAT) sends a command to the ECU and the ECU responds with the data.

We've found that on top of the CANbus protocol, the ECU's add ISOTP packetisation. This allows the ECUs to communicate in larger than 8 byte frames. At the lowest level 8 byte packets are sent but with ISOTP on top it can bunch up multiple 8 byte packets into ISOTP frames - allowing larger transmission chuncks to/from the application layer to the car.

I've called this a "ISOTP-like" protocol because it uses the packet structure of ISOTP but critically seems to omit the use of FlowControl. So, we've had to modify some standard Python libraries to be able to talk to the car.

We haven't made a whole lot of communications to our cars yet, but we have sent commands to the cars and got responses for things like Serial Number, Engine RPM and Coolant Temp.

On top of the ISOTP protocol the ECUs are adding another control layer. We don't completely understand this layer at the moment but we are able to "hard code" some commands and then get responses. We have seen a couple of these commands so far, one that asks the car for its serial number and another that asks for data back from the car.

The second of these commands is also interesting in that it can ask for multiple data items back from the car at once. It can ask for single items but it can also ask for multiple items and so make the communication on the bus much more efficient.

In parallel with this work around the commands, we've also spent a lot of time trying to figure out what each data request means and what the respective results are - how the data is placed in the response, how it is scaled and what its ranges are. To that end we've decoded the Easimap EC2 file used for our ECUs (9A4be52a.ec2). As a result we have a Python tool now that can take an EC2 file and extract the communication protocol used by Easimap and store it in a form that we can now write software against (we have it in csv, json and python formats).

Here's an example of the configuration for engine rpm...

    {
        "name" = "RT_ENGINESPEED".
        "page" = "0xF8".
        "address" = "0x237C".
        "bytes" = "2".
        "scale_minimum" = "0.000000".
        "scale_maximum" = "65535.000000".
        "display_minimum" = "0.000000".
        "display_maximum" = "12000.000000".
        "display_interval" = "1000.000000".
        "units" = "RPM".
        "short_desc" = " Engine Speed".
        "long_desc" = " Engine Speed in rpm as measured by the crankshaft sensor. This is the primary input for the fuel and ignition maps"
    }

The page and address fields above are used to create the "request for data" commands and then the remainder of the configuration tells us how to interpret the results. The "bytes" field tells us how many bytes of data will come back from the car.

If anyone is interested to know more then please contact CtrMint to join our Slack workspace or DM me and I can send more info. The Slack channel has all the source code and output files that we've been working on.

I must stress that this is all a work-in-progress. Any of this information may change... though we think (hope) we are on the right track.

I hope that's vaguely interesting to a few of you. More to come over the next few days/weeks... and there will be a full write-up, source code and sample files available on the subject at some point.

John, Mark and James.

[ECR]

This is a strangely addictive thread (even though I understand NONE of it!). It's very satisfying seeing three people in different locations, none of whom knew each other before (?), forming an alliance to try to crack a problem. I'll continue to be fascinated...

Good Luck  

[Jonathan Kay]

Well said.

I only understand a bit, but it doesn't decrease the enjoyment.

Jonathan

[Purplemeanie]

First off, thanks to all those that have shown some interest in this thread, it can get a bit technical at times, so shout if you'd like me (us) to go into more detail. Or shout if you want me to stop :-)

Ok, so another update. 

Summary

We now understand the communications protocol used by Easimap and can use it to ask the ECU for whatever data the car has to offer. We can then manipulate the responses we get back from the car and format it to make sense to us.

If you don't want the technical jargon, then skip to the bottom of the post for the Results :-)

TL;DR

I've got to a point now where I think we understand how Easimap is talking to the car and how it formats the data fields it gets from the car. And therefore we can just do the same to get whatever data the car has to offer.

What I haven't looked at yet is how Easimap can set parameters in the car's ECU. I will take a look at that at some point but for the moment my interest is in creating some sort of Linux tool that can extract data from the car and display it. I may also have a look at whether the mappings on the ECU can be changed, though the general consensus is that the ECU's are locked, and I have no reason to change that theory at the moment. I'm not sure how that locking occurs or whether it can be circumvented. At the moment I'm not interested in that side of things and have no intention of looking at it - at least at the moment.

So, to that end... here's what I understand now about what I called in the previous post the "MBE ISOTP-like" protocol used by MBE in their Easimap software and the ECU's.

Firstly, this protocol is a multi-layered protocol. It's a bit like Russian Dolls, the actual data packets used by MBE are a couple of layers deep inside the doll. In order to look at the data the way that the ECU does we have to capture the data from the OBD port and then unpack it a couple of times to get at the real data.

The unpacking layers in this case are CAN, ISOTP (ISO-15765) and a proprietary MBE layer.

Lets dig into the various layers and explain what's going on, hopefully this will help future developers. I also intend to do a full writeup on my blog and I'm sure Mark will do the same for what he's been looking at. I'll have diagrams and code snippets there to help. Unfortunately, this L7 website is a little limited in the sorts of diagrams and files I can show you here, so my website will be a better place for a full write-up. I'll post a link when it goes live.

CAN Layer

The OBD-II port (on my car (2017 420R) and most modern-ish Caterhams) connects to the ECU by way of a two wire differential voltage protocol. It uses CAN-H and CAN-L to signal 1s and 0s on the CAN bus.

There are two critical things here in each CAN packet... firstly there's the CAN ID (identifier) and then there's the Data. The CAN ID can be either 11bits long or 29bits long. On my car with the MBE 9A4 ECU, it uses 29bit IDs. But other Caterhams also seem to use 11bit, probably with unlocked ECUs.

Secondly, the CAN packet contains up to 8 bytes of data. The CAN standard does allow for more than 8 bytes but it doesn't seem as though our cars use that version of the protocol.

Other than the ID and data the CAN packet also contains lots of start bits, CRC (cyclic redundancy check) error checking, data length parameters and an ACK section. The ACK section is perhaps worth a few words: a sending device on the network only knows if "something" has received it's CAN packet if a receiver sets the ACK bit when it received the data.

Anyway, on to the more interesting stuff.

The 29bit CAN IDs are used a payload type identifier - When a device sending on the bus sets a particular ID, it is telling any receiver what sort of protocol to expect in the packet.  I'll talk more about the specifics of the IDs used in our cars when I get onto the ISOTP layer below.

So, at a high level view of each CAN packet, there are the two things we're interested in, the CAN ID and the Data. For our cars the IDs are 29bit and the data can be up to 8 bytes. And a packet looks like this when we've captured it from the CAN bus:
  #   Time        Prot Len Info CAN ID       Data
  168 1.612770860 CAN  32  XTD: 0x0cbe1101   10 08 01 00 00 00 00 f8
  169 1.613462515 CAN  32  XTD: 0x0cbe1101   21 7c 7d 00 00 00 00 00
  170 1.614548554 CAN  32  XTD: 0x0cbe0111   03 81 34 12 00 00 00 00
  
This data is laid out in columns by the packet sniffing tool I'm using (Wireshark/tshark). 
- # = Packet Number
- Time = number of seconds since the packet capture started
- Prot = Protocol (CAN)
- Len  = Length of the capture packet. This is not the CAN data length, that's always 8 for us
- CAN ID = 29bits (shown as hexadecimal in this example)
- Data = Data in the CAN packet, shown as hexadecimal bytes

You'll see why I picked those particular packets a bit later.

So, to recap... when we put a packet sniffer (Raspberry Pi with PICAN2 interface in my case) and watch (sniff) the CAN bus, we see a succession of lines of output from our packet sniffer like above... 

ISOTP

Ok. So lets look at one level of Russian Dolls down (smaller doll).

As you can see from the example above, our particular implementation of the CAN protocol allows the car and any other device on the CAN bus to talk in chunks of 8 bytes at a time.

Now, this is quite restrictive for a modern car. The communication between Easimap and the car ECU might need to send all sorts of data that is longer than 8 bytes. For instance a 2 dimensional engine fueling map that might be hundreds of bytes long.

The MBE ECU and Easimap get over this problem by adding another protocol on top of the CAN interface (another Russian Doll). This protocol is called ISO-15765 or ISO-TP. By formatting the data in the CAN packets we can send one ISO-TP "frame" across multiple CAN packets - allowing us to send more than 8 bytes in a frame.

Here's the same 3 CAN packets from above, but this time we looking at it through an ISO-TP dissector (its dissecting the packet stream and viewing it as though it is formatted in accordance with the ISO-TP protocol).

  #   Time        Proto    Len Info                       Data
  168 1.612770860 ISO15765 32 First Frame(Frame Len: 8)   01 00 00 00 00 f8
  169 1.613462515 ISO15765 32 Consecutive Frame(Seq: 1)   7c 7d
  170 1.614548554 ISO15765 32 Single Frame(Len: 3)        81 34 12

Packets 168 and 169 are actually a single ISO-TP frame. The "First Frame" and the "Consecutive Frame" are "fragments" of a whole frame and can be joined together to form an ISO-TP frame with the following data:

  01 00 00 00 00 f8 7c 7d
  
The "Frame Len: 8" in the First Frame tells us there are 8 bytes of data at the ISO-TP protocol layer.

Now, you might say... but hang on, we could send 8 bytes of data with just CAN frames. Well, that's true in this particular example but for more complex examples we can send over 4000 bytes of data in a single ISO-TP frame (using many CAN packets).

One significant departure that MBE seem to make from the ISO-TP standard is that they do not implement flow-control in the protocol. We don't need to go into that here, but if anyone is trying to talk to an MBE ECU using their own software, then they'll need to remove any notion of flow control from the ISO-TP layer they use. At least that's my current theory - and it seems to be stacking up.

Right, so now we've used up two layers of the Russian Doll. The CAN layer and the ISO-TP layer. In the third layer (I'll call it the MBE layer) we see exactly what the ECU and Easimap are really saying to each other.

MBE Layer

The data we extracted using the ISO-TP dissector has given us the following (same as above):

  01 00 00 00 00 f8 7c 7d
  
What does that mean to the ECU?

Well, this took a bit of figuring out. We don't have MBE's complete rule-book for decoding this layer of the communications. But what we do have is their Easimap software. That helps us in two ways. Firstly, we can watch Easimap and the ECU talking to each other. That's how we got these sample packets. But that just looks like random data at the moment. Secondly, we have the EC2 data files that Easimap loads when it discovers what ECU its talking to. MBE supports a whole raft of ECU's and each seems to communicate with Easimap in a different way. MBE could have coded each of those protocols into Easimap but that would mean that they needed to update the Easimap software whenever they make a new ECU for a customer (Caterham or whoever).

So, MBE wrote Easimap to discover what ECU its talking to and then load the correct EC2 file for that ECU. Fortunately for us the EC2 files are easily found in the Easimap installation.

A Digression into EC2 files

The EC2 files are worth a whole post in themselves. And we had to write scripts to decode them and give us the keys to unlock the MBE layer.

In simple terms though, the EC2 files set up a lot of "variables". Each variable is a type of data that Easimap can extract from the ECU. The 9A4 EC2 file that Easimap uses for our cars has over 2000 variables defined. It turns out that over half of them are "disabled" but that still leaves over 900 to work with. I suspect that many of these 900 don't actually do anything with our cars but I'm planning to got and check which are "active" and which are not.

For each variable in the EC2 file there a definitions for things like, the number of bytes of data each variable takes up in the MBE communications layer, the short and long description of the variable that Easimap uses when showing the data on the screen and crucially it also has entries for what it calls a "page" and an "address".

This information for each variable is spread all over the EC2 file and the scripts we wrote had to find each variable and pull the respective information together to build a picture of how each variable works.

{'address': '0x237c',
 'bytes': '2',
 'display_interval': '1000.000000',
 'display_maximum': '12000.000000',
 'display_minimum': '0.000000',
 'long_desc': ' Engine Speed in rpm as measured by the crankshaft sensor. This '
              'is the primary input for the fuel and ignition maps',
 'name': 'RT_ENGINESPEED',
 'page': '0xf8',
 'scale_maximum': '65535.000000',
 'scale_minimum': '0.000000',
 'short_desc': ' Engine Speed',
 'units': 'RPM'}
 
[ You can read the info as address=237c etc. ]

Once we'd decoded the EC2 file we could start to try and guess on the protocol used in this MBE layer.

Back to the MBE Layer

I picked all this example data and info for a reason. I had suspected ever since looking at the EC2 files for the first time a few weeks ago that the page and address info was important. And I had one of thos Eureka moments when I looked at the output of the ISO-TP layer and realized from:

  01 00 00 00 00 f8 7c 7d

The f8 looked like a page from the EC2 file (see 'page' : '0xf8' above) and the 7c looked like part of the address. So, I went away and hand decoded a bunch of messages from the ECU and low and behold it seemed to match up. So page=f8 and address Low-Byte (the 7c bit of the address) meant RT_ENGINESPEED - the engine RPMs.

The next thing was to figure out what the rest of the packet means.

And so I wrote a bunch more code and started to test out my theories.

After the [01 00 00 00 00 f8 7c 7d] message is sent by Easimap, the ECU responds with:

  81 34 12

One of the ways that we saw both CAN and ISO-TP communicate was to send a "command" in the first byte of the packet/frame. And for the response to have a similar format but with one or more other bits in the response byte being set.

So, it seemed to make sense that the initial 01 byte in the frame from Easimap was responded to by the ECU with a response byte of 81 (81 in hexadecimal is 01 with the top bit set). This was starting to add up and so the request [01 .... 7d] was a data request for RT_ENGINESPEED and the ECU responded with 3412 (after the 81).

Now I've connived to make the data here readable. When I was first testing I didn't have the car running and so the ECU was responding with 81 00 00... meaning 0 RPM. But when I started up the car the responses turned into real engine rev data.

Turns out that the [34 12] is sent in reverse order and the bytes need to be swapped, making the engine revs 0x1234, which in decimal are 4660. That's revving the car a little highly in a test, but you can see how the data works.

Addresses are Weird

One of the complications around access to the ECU seems to be some weird mapping going on between the EC2 file and the address requests going to the ECU. For instance, if we take our RT_ENGINESPEED variable then the EC2 address is 0x237c.

But what Easimap sends to the ECU is a request for two bytes, a Lowest Significant Byte (LSB) and a Most Significant Byte (MSB) or 0x7c and 0x7d i our RT_ENGINESPEED example.

Having tried this out with a few dozen different variables it seems that Easimap takes the LSB of the Easimap address (0x7c in this example) and increments (adds one) for each extra byte it wants from the ECU. 

Easimap sometimes asks for 4-byte data and so this incrementing can be seen here, for example, with RT_ENGINERUNTIME...

This is the variable info for RT_ENGINERUNTIME
{'address': '0xcfcecdcc',
 'bytes': '4',
 'display_interval': '10000.000000',
 'display_maximum': '1193046.500000',
 'display_minimum': '0.000000',
 'long_desc': ' Engine Run Time',
 'name': 'RT_ENGINERUNTIME',
 'page': '0xe2',
 'scale_maximum': '1250999.875000',
 'scale_minimum': '0.000000',
 'short_desc': ' ERT',
 'units': 'Hours'}
RT_ENGINERUNTIME=3.9094e+04 Hours ( ERT ) [0x07ffffff=134217727, Scale:1250999.875, Div:4294967295, Offset:0.0]

But that's a bit weird too, because the EC2 address ('0xcfcecdcc') is the actual series of requests sent to the ECU. I'm a little baffled at the moment about when Easimap uses the EC2 address and when it makes one up. I suspect there are rules depending on how many bytes are being asked for.

Compound Data Requests

Lets talk about another twist to the tale. In order that Easimap can talk to the ECU in an as-efficient way as possible. It batches up requests to pages inside the ECU and asks for a bunch of data results in one go. A request and response will therefore look like this at the CAN layer:

   46 0.353164720                             CAN 32 XTD: 0x0cbe1101   10 23 01 00 00 00 00 f8
   47 0.353978868                             CAN 32 XTD: 0x0cbe1101   21 30 31 36 37 44 45 4c
   48 0.354710862                             CAN 32 XTD: 0x0cbe1101   22 4d 4e 4f 50 51 5a 5b
   49 0.355482989                             CAN 32 XTD: 0x0cbe1101   23 5c 5d 64 6a 6b 7c 7d
   50 0.356219206                             CAN 32 XTD: 0x0cbe1101   24 9e 9f a0 a1 d8 d9 da
   51 0.356950718                             CAN 32 XTD: 0x0cbe1101   25 db 9f a0 a1 d8 d9 da
   52 0.358137040                             CAN 32 XTD: 0x0cbe0111   10 1e 81 4e 39 c7 47 e0
   53 0.358462988                             CAN 32 XTD: 0x0cbe0111   21 54 d4 83 3a 85 4e 39
   54 0.358805252                             CAN 32 XTD: 0x0cbe0111   22 a8 0d a2 a0 00 b0 4f
   55 0.359140831                             CAN 32 XTD: 0x0cbe0111   23 00 00 7a 9c 68 08 00
   56 0.359467705                             CAN 32 XTD: 0x0cbe0111   24 80 00 80 a1 d8 d9 da

Which then looks like this at the ISO-15765 layer:

   46 0.353164720                             ISO15765 32 First Frame(Frame Len: 35)   01 00 00 00 00 f8
   47 0.353978868                             ISO15765 32 Consecutive Frame(Seq: 1)   30 31 36 37 44 45 4c
   48 0.354710862                             ISO15765 32 Consecutive Frame(Seq: 2)   4d 4e 4f 50 51 5a 5b
   49 0.355482989                             ISO15765 32 Consecutive Frame(Seq: 3)   5c 5d 64 6a 6b 7c 7d
   50 0.356219206                             ISO15765 32 Consecutive Frame(Seq: 4)   9e 9f a0 a1 d8 d9 da
   51 0.356950718                             ISO15765 32 Consecutive Frame(Seq: 5)   db 9f a0 a1 d8 d9 da
   52 0.358137040                             ISO15765 32 First Frame(Frame Len: 30)   81 4e 39 c7 47 e0
   53 0.358462988                             ISO15765 32 Consecutive Frame(Seq: 1)   54 d4 83 3a 85 4e 39
   54 0.358805252                             ISO15765 32 Consecutive Frame(Seq: 2)   a8 0d a2 a0 00 b0 4f
   55 0.359140831                             ISO15765 32 Consecutive Frame(Seq: 3)   00 00 7a 9c 68 08 00
   56 0.359467705                             ISO15765 32 Consecutive Frame(Seq: 4)   80 00 80 a1 d8 d9 da

Which is a request like this at the MBE layer:
  01 00 00 00 00 f8 30 31 36 37 44 45 4c 4d 4e 4f 50 51 5a 5b 5c 5d 64 6a 6b 7c 7d 9e 9f a0 a1 d8 d9 da db

Followed by a response of this:
  81 4e 39 c7 47 e0 54 d4 83 3a 85 4e 39 a8 0d a2 a0 00 b0 4f 00 00 7a 9c 68 08 00 80 00 80
  
Or once the MBE layer is decoded it's a request for:

This is a command request for data in page: f8 ...
[{'bytes': '2', 'name': 'RT_THROTTLEANGLE1(RAW)'},
 {'bytes': '2', 'name': 'RT_AIRTEMP1(LIM)'},
 {'bytes': '2', 'name': 'RT_COOLANTTEMP1(LIM)'},
 {'bytes': '2', 'name': 'RT_COOLANTFUELFACTOR'},
 {'bytes': '2', 'name': 'RT_AIRTEMPFUELFACTOR'},
 {'bytes': '2', 'name': 'RT_THROTTLEANGLEINCREASING'},
 {'bytes': '2', 'name': 'RT_TPSFUEL+TRIMBANK1'},
 {'bytes': '2', 'name': 'RT_TPSVSSPEEDIGN+TRIM1'},
 {'bytes': '1', 'name': 'RT_THROTTLESITE1'},
 {'bytes': '2', 'name': 'RT_BAROSCALEDLIM'},
 {'bytes': '2', 'name': 'RT_ENGINESPEED'},
 {'bytes': '2', 'name': 'RT_BATTERYVOLTAGE(LIM)'},
 {'bytes': '2', 'name': 'RT_BATTERYVOLTAGECOMP'},
 {'bytes': '2', 'name': 'RT_MAPPINGPOT1LIM'},
 {'bytes': '2', 'name': 'RT_MAPPINGPOT2LIM'}]
 
and a response with these values:

RT_THROTTLEANGLE1(RAW)=1.1192 V ( Throttle Angle 1 (Raw) ) [0x394e=14670, Scale:5.0, Div:65535, Offset:0.0]
RT_AIRTEMP1(LIM)=14.862  ( Air Temp ) [0x47c7=18375, Scale:160.0, Div:65535, Offset:-30.0]
RT_COOLANTTEMP1(LIM)=23.048  ( Coolant Temp ) [0x54e0=21728, Scale:160.0, Div:65535, Offset:-30.0]
RT_COOLANTFUELFACTOR=105.98  ( Coolant Fuel Factor ) [0x83d4=33748, Scale:400.0, Div:65535, Offset:-1e+02]
RT_AIRTEMPFUELFACTOR=4.0848  ( Air Temp Fuel Factor ) [0x853a=34106, Scale:200.0, Div:65535, Offset:-1e+02]
RT_THROTTLEANGLEINCREASING=1.1192 V ( Throttle Angle Increasing ) [0x394e=14670, Scale:5.0, Div:65535, Offset:0.0]
RT_TPSFUEL+TRIMBANK1=5.6013 ms ( TPS Fuel + Trim ) [0x0da8=3496, Scale:105.0, Div:65535, Offset:0.0]
RT_TPSVSSPEEDIGN+TRIM1=4.7022  ( TPS vs Speed Ign + Trim ) [0xa0a2=41122, Scale:-120.0, Div:65535, Offset:80.0]
RT_THROTTLESITE1=0.0 Site ( Throttle Site 1 ) [0x00=0, Scale:16.0, Div:255, Offset:0.0]
RT_BAROSCALEDLIM=1.04 Bar ( Baro Pressure ) [0x4fb0=20400, Scale:6.5535, Div:65535, Offset:-1.0]
RT_ENGINESPEED=0.0 RPM ( Engine Speed ) [0x0000=0, Scale:65535.0, Div:65535, Offset:0.0]
RT_BATTERYVOLTAGE(LIM)=12.225 V ( Battery Voltage ) [0x9c7a=40058, Scale:20.0, Div:65535, Offset:0.0]
RT_BATTERYVOLTAGECOMP=0.43116 ms ( Battery Voltage Comp ) [0x0868=2152, Scale:13.13, Div:65535, Offset:0.0]
RT_MAPPINGPOT1LIM=0.0015259  ( Mapping Pot 1 ) [0x8000=32768, Scale:200.0, Div:65535, Offset:-1e+02]
RT_MAPPINGPOT2LIM=0.0015259  ( Mapping Pot 2 ) [0x8000=32768, Scale:200.0, Div:65535, Offset:-1e+02]

All that done in one ISO-TP request frame and one response frame.

Results

Most of the variables have scale and offset factors that have to be applied, but initial results give the following for the default Easimap screen...

RT_CURRENTFAULTSB=3.2769e+04 - ( Current Faults B ) [0x8001=32769, Scale:65535.0, Div:65535, Offset:0.0]
RT_ENGINE_CONFIG=176.0 - ( Eng Cnfg ) [0xb0=176, Scale:255.0, Div:255, Offset:0.0]
RT_MEMORYCONFIG=64.0 - ( Memory Configuration ) [0x40=64, Scale:255.0, Div:255, Offset:0.0]
RT_IGNITIONADVANCEBANK1=23.691  ( Ignition Advance ) [0x7820=30752, Scale:-120.0, Div:65535, Offset:80.0]
RT_BAROFUELCOMP=3.5294  ( Barometric Pressure Fuel Compensation ) [0x84=132, Scale:200.0, Div:255, Offset:-1e+02]
RT_CRANKCOUNT=463.0 - ( Crank Count ) [0x01cf=463, Scale:65535.0, Div:65535, Offset:0.0]
RT_ENGINERUNTIME=3.9094e+04 Hours ( ERT ) [0x07ffffff=134217727, Scale:1250999.875, Div:4294967295, Offset:0.0]
RT_THROTTLEANGLE1(RAW)=1.5767 V ( Throttle Angle 1 (Raw) ) [0x50ba=20666, Scale:5.0, Div:65535, Offset:0.0]
RT_AIRTEMP1(LIM)=14.615  ( Air Temp ) [0x4762=18274, Scale:160.0, Div:65535, Offset:-30.0]
RT_COOLANTTEMP1(LIM)=23.121  ( Coolant Temp ) [0x54fe=21758, Scale:160.0, Div:65535, Offset:-30.0]
RT_COOLANTFUELFACTOR=22.719  ( Coolant Fuel Factor ) [0x4e8a=20106, Scale:400.0, Div:65535, Offset:-1e+02]
RT_AIRTEMPFUELFACTOR=4.0848  ( Air Temp Fuel Factor ) [0x853a=34106, Scale:200.0, Div:65535, Offset:-1e+02]
RT_THROTTLEANGLEINCREASING=1.5767 V ( Throttle Angle Increasing ) [0x50ba=20666, Scale:5.0, Div:65535, Offset:0.0]
RT_TPSFUEL+TRIMBANK1=4.5887 ms ( TPS Fuel + Trim ) [0x0b30=2864, Scale:105.0, Div:65535, Offset:0.0]
RT_TPSVSSPEEDIGN+TRIM1=24.32  ( TPS vs Speed Ign + Trim ) [0x76c8=30408, Scale:-120.0, Div:65535, Offset:80.0]
RT_THROTTLESITE1=8.2824 Site ( Throttle Site 1 ) [0x84=132, Scale:16.0, Div:255, Offset:0.0]
RT_BAROSCALEDLIM=1.04 Bar ( Baro Pressure ) [0x4fb0=20400, Scale:6.5535, Div:65535, Offset:-1.0]
RT_ENGINESPEED=2400.0 RPM ( Engine Speed ) [0x0960=2400, Scale:65535.0, Div:65535, Offset:0.0]
RT_BATTERYVOLTAGE(LIM)=13.087 V ( Battery Voltage ) [0xa782=42882, Scale:20.0, Div:65535, Offset:0.0]
RT_BATTERYVOLTAGECOMP=0.33138 ms ( Battery Voltage Comp ) [0x0676=1654, Scale:13.13, Div:65535, Offset:0.0]
RT_MAPPINGPOT1LIM=0.0015259  ( Mapping Pot 1 ) [0x8000=32768, Scale:200.0, Div:65535, Offset:-1e+02]
RT_MAPPINGPOT2LIM=0.0015259  ( Mapping Pot 2 ) [0x8000=32768, Scale:200.0, Div:65535, Offset:-1e+02]
RT_SOFTCUTTIME=7800.0 RPM ( Soft Cut ) [0x1e78=7800, Scale:65535.0, Div:65535, Offset:0.0]
RT_HARDCUTTIME=7900.0 RPM ( Hard Cut ) [0x1edc=7900, Scale:65535.0, Div:65535, Offset:0.0]
RT_INJECTIONTIMEA=7.2051 ms ( Final Injection Time ) [0x1191=4497, Scale:105.0, Div:65535, Offset:0.0]
RT_DUTYCYCLEA=14.51  ( Duty Cycle ) [0x25=37, Scale:100.0, Div:255, Offset:0.0]
RT_HAVESPEEDS=169.0 - ( RT_HAVESPEEDS ) [0x00a9=169, Scale:65535.0, Div:65535, Offset:0.0]

Some of this still isn't making sense but most of it seems to.

I hope that's been at least somewhat informative for people. Feels like I should do a lecture course on the subject [ and really put you all to sleep ] :-)

John

[CtrMint]

John's added some excellent detail there. 

We've also now gained control over Easimap, permitting a test harness to be coded and implemented.  This will enable testing and validation of ECU information.  It will be particularly handy when it comes to sanity checking our interpretation of values for an SBC logging solution or similar.

The image illustrates the control having sent an intentionally large RPM value at Easimap while running, isolated from the car.

[Toughie]

And I thought the Kerbal Space Programme could get technical!

 

.

[Purplemeanie]

Hi Toughie... I hope that's a vote of confidence in what we're doing :-)

All these shenanigans are (I'm afraid) bread and butter for some of us doing this work. In the words of the slogan that was behind the Wireshark network sniffer a few years ago... we spend our working lives "Sniffing the glue that holds the internet together"! Packet analysis on an OBD canbus should be relatively simple, and certainly isn't rocket science ;-)

New post coming later tonight...

[Purplemeanie]

Progress report...

Up until now, a lot of what we posted about the MBE ECU protocols has been a little theoretical. We've sniffed away at the packets on the canbus and we've poked and prodded at Easimap. We also wrote some simple test applications to test out our theories.

But now we have a software tool to ask a standard locked MBE 9A4 to send us whatever data we want to ask it for. We've now been able to select which data parameters we want to know about and for the car to respond. We can then repeatedly ask the car for updates as often as we dare and to produce things like dashboards, data-loggers and change lights etc.

I run these tools run on a Raspberry Pi at the moment but should work on any Linux system. It uses a PiCAN2 to access the canbus, but could use any SocketCAN compatible device.

The code is available on GitHub for anyone to download, look at, and throw stones at, if that's what you're into. The code can be found here:

https://github.com/Purplemeanie

For the softies amongst you: The can-isotp repository is a modified kernel module that can knoble flow-control in the ISOTP protocol layer. The Caterham-OBD repo has the main access class (mbe.py) and a test app. There are also other support tools and files in there too.

The code is still quite rough and ready, so don't be too critical of it.... it's a work in progress.

So... by way of an example: I asked the car for the following EC2 data variables:

RT_AIRTEMP1(LIM)
RT_COOLANTTEMP1(LIM)
RT_ENGINESPEED
RT_BATTERYVOLTAGE(LIM)
RT_SOFTCUTTIME
RT_HARDCUTTIME

And got the following back. As you can see, the battery needs charging and the engine wasn't running!...        

Air Temp	RT_AIRTEMP1(LIM)	16.189669642175936 C
Battery Voltage	RT_BATTERYVOLTAGE(LIM)	11.815976195925842 V
Coolant Temp	RT_COOLANTTEMP1(LIM)	21.95635919737545 C
Engine Speed	RT_ENGINESPEED	0.0 RPM
Hard Cut	RT_HARDCUTTIME	7900.0 RPM
Soft Cut	RT_SOFTCUTTIME	7800.0 RPM

If needed I can ask for that data repeatedly and get a continuous readout showing "real time" values for all the data points I ask for.

I think we can say we have the basics now for a set of tools to be able to write whatever applications we want.

Not bad progress for a few weeks tinkering. I'm just waiting for some old-hand to pop up and say they did this years ago. But what the heck, its been great fun solving the puzzle.

John

[John Vine]

Amazing stuff!  Although I barely understand it, it does come across as highly relevant.

I take it you can't get at the map?

JV

[Purplemeanie]

Hi John,

  I haven't tried to extract a map yet. I suspect it needs a different approach than the one we've been using to get the ECU variables. I'm also not that hopeful that we'll succeed, they are supposed to be locked after all and what we've done so far isn't anything that MBE have specifically protected against. They haven't used any encryption or subterfuge that we've had to break through, they've just used already existing protocols with a couple of twists. It wasn't as though they were hiding the info, it just took some persistence to figure out what they were doing. However, these ECUs are supposed to be locked and so I suspect won't give up a map that easily.

  Attempting to read a map is on the list of things to have a look at though.

John

[ECR]

As John says, amazing ......

[CtrMint]

A quick example of a possible tool.

[CtrMint]

Things are getting close. 

Just need to finalize a few points on the comms side and we should have a first pass at a replacement for Easimap when used for diagnostic work.  If all goes to plan an SBC such as a Raspberry Pi can be fitted to the car and live data transferred wireless to the following dashboard.

Random test data shown

[CtrMint]

This evening I believe I've completed the dashboard I've been working on.  I've attached a quick video illustrating the first pull of live data from the car.

I appreciate the font isn't great, but that's a quick fix.  Unfortunately the font I'd used during development isn't available on the SBC (Raspberry Pi).

You'll probably also note the values shown are all integers, again a small thing to fix.  If you look at the data shown on the command line / stdout towards the end of the video, you'll see floating points/real numbers are available.  I just need to pack them into data flow.

It was late when this was recorded, and so I couldn't start the car, thus no RPM shown.  I'll start the car tomorrow and capture that.  It's way cooler, moving dials etc.  Though I wouldn't bring the SHIFT symbol on for obvious reasons.

For those that might be interested, I was using a virtualized Raspberry Pi, OS (Raspbian) running on a Macbook Air, this was then connected to the car using a Socketcan compatible device, a Korlan USB2CAN.  No Easimap cable needed.  This setup will be replaced with a physical Raspberry Pi.

For the real geek stuff now.

It's a client-server model. There's two distinct programs, a client and a server.   The server displays the dash, and listens for network data being sent, currently over the loopback interface.  The server decodes received data and displays as received.  Table labels and values are created based on the data sent, so it's very flexible.  The client talks to the ECU, pulls the values, packages them and transmits the data over the network, again currently via the loopback.   I'm using UDP for ease and thus connectionless comms.  Going forwarded I'll be able to fit an infrastructure AP to the Pi which is installed in the car, ask it to run the client, and then use the server/dash on a Wifi connected laptop.  This will enable use of the Dash and review of the data without the need for a physical connection.

Hope that's still of interest 

credit to John for his work putting together a class to convert the ecu vals.

[Purplemeanie]

Very cool Mark!

[Jonathan Kay]

My limited understanding might be about to show.

You can't (yet) get the map as a data object.

But could you deduce all of the production rules that make up the map by injecting the relevant input values and recording the outputs? And with the work you've done already could that be automated?

Thanks

Jonathan

[Purplemeanie]

Hi Jonathan,

  In theory, yes.

  If we can see all the inputs to the ECU (and I think we can) and we can see all the outputs (again, I think we can) then as we drive the car around (or perhaps more scientifically on a rolling road) we should be able to deduce the map - especially as we know how many maps there are (from the unlocked ECU maps that are around) and what format they take.

  ... in theory.

  It shouldn't be too difficult to write some software to do exactly that and test your suggestion.

  While Mark is working on his dashboard, I'm working on a steering-wheel-computer (more on that later). Once I'm done with that, and my blog updates, then I'll have time to do some testing. Now we have the basic code, it should also be possible for another softie to take what we've done and test your theory.

  BTW... if anyone needs more help with their insomnia, or found my previous posts on this thread too light weight, then I've started to write all the gorey detail out on my blog: https://www.purplemeanie.co.uk/index.php/2019/08/31/ecu-diagnostics-part-1-introduction/

John

[Purplemeanie]

As a teaser of things to come... here's a render of my next project to create a Steeting Wheel Computer for my car. It will connect to the CAN bus Raspberry Pi using Bluetooth and show stats and shift-lights on the steering wheel...

The render shows the wheel and electronics... I'll "wrap" the electronics in a plastic case that I'll 3D print.

John