Post Reply 
PS1 CD-ROM EDC and ECC
06-24-2017, 11:46 PM
Post: #21
RE: PS1 CD-ROM EDC and ECC
MysticLord how's your progress with this EDC and ECC? Were you able to get some help from AbyssionM? It's been a while and I've fell off the wagon with this.
Find all posts by this user
Quote this message in a reply
09-27-2017, 04:44 AM
Post: #22
RE: PS1 CD-ROM EDC and ECC
I'm working on a test program to see if I can generate valid EDC and ECC in Mode 2, Forms 1 and 2. Loaded the 2 tables from a text file (originally found here ). Here are my notes so far.

Quote:cdrom_ecc.py

crc32(data) is used in uniso.py

encode_L2_P(data) is not found in randomtools-master but for cdrom_ecc.py, but is used in get_edc_ecc(data)

encode_L2_Q(data) ^same as above^

get_edc_ecc(data) is found in:
* psx_file_extractor.py in write_data_to_sectors(imgname, initial_sector, datafile="_temp.bin")
* uniso.py in inject_logical_sectors(sourcefile, outfile, debug=False) at lines 72 and 85

get_edc_form2(data) is found in cdrom_ecc.py

walk through the code where "data" is defined to figure out what it is.

My next step is to figure out what version of python this is - if that matters - then look up the code used to define the data parameters to figure out what they are. After that I translate the python code to Java, test it, and debug it until it works.

When I'm done I should have a bit of code for doing disc image i/o on any PS1 game that doesn't use custom LBA (logical byte addressing) methods to delimit and locate files on a disc image. I could probably add in code to handle LBAs and have the user define this in various configuration files. Once I have that there's no point in not adding methods to extract files and folders, insert files and folders, change their sizes, and move them around. I could also specify if a file is compressed and if so, what method should be used to de/compress it. Granted the person who makes an editor will need to implement that themselves, but once the functionality exists there it's there forever. The basics of dealing with disc images is the same for all types of disc images, the specifics won't change a lot between cd-roms, dvds, and blue-rays.

I hope this brings a new era of cross-platform editors for disc image based games.
Find all posts by this user
Quote this message in a reply
09-27-2017, 09:37 PM
Post: #23
RE: PS1 CD-ROM EDC and ECC
Quote:1. Is the Randomtools repo on your github is everything you need to do EDC/ECC?
The most recent version should be, yeah. If it has a file called psx_file_extractor.py, the EDC/ECC handling should be spot on.

Quote:2. What is the "data" parameter in the methods found in cdrom_ecc.py? An integer array?
It's a list, not an array, of mostly integers with some "None" values thrown in. The first 2064 values are the bytes to encode. The remaining values all start as "None" until they are gradually filled in with the bytes of the ECC by encode_L2_P and encode_L2_Q.

Quote: encode_L2_P(data) is not found in randomtools-master but for cdrom_ecc.py, but is used in get_edc_ecc(data)
It's in line 20 of cdrom_ecc.py, the same file as get_edc_ecc.

Quote:My next step is to figure out what version of python this is
Anything 2.x should work, but I use 2.7.

Quote:After that I translate the python code to Java, test it, and debug it until it works.
I should reiterate that this code, itself, was already ported from C. Same function names, etc. I suppose I can't blame you for wanting to port it to your preferred language, because that's exactly what I did, but you might find that the C version is closer to what it would look like in Java.

Quote:When I'm done I should have a bit of code for doing disc image i/o on any PS1 game that doesn't use custom LBA (logical byte addressing) methods to delimit and locate files on a disc image.
I've messed around with this. Run psx_file_extractor.py on a disk image and it should export the file tree. This module also contains tools for importing the modified files and generating the correct ECCs to match. I've been using it with some success on a few particular games.
Find all posts by this user
Quote this message in a reply
09-30-2017, 09:30 AM
Post: #24
RE: PS1 CD-ROM EDC and ECC
(09-27-2017 09:37 PM)abyssonym Wrote:  
Quote:2. What is the "data" parameter in the methods found in cdrom_ecc.py? An integer array?
It's a list, not an array, of mostly integers with some "None" values thrown in. The first 2064 values are the bytes to encode. The remaining values all start as "None" until they are gradually filled in with the bytes of the ECC by encode_L2_P and encode_L2_Q.

(04-12-2017 06:16 AM)abyssonym Wrote:  I can describe what the tool does in more detail, though. Basically the main function is get_edc_ecc which handles Mode 2 Form 1 sectors exclusively. This function was ported from one case of the VCD Imager function do_encode_L2. My version, get_edc_ecc, takes a 2072-byte data string as a parameter and returns a 4-byte error detection code (EDC) and a 276-byte error correction code (ECC). These three strings, when concatenated, produce one 2352-byte sector.

Mode 2 Form 2 sectors are handled differently though. These sectors omit the ECC field so they have an extra 276 bytes for data. Most playstation games have a mixture of both Form 1 and Form 2 sectors on them. Form 2 sectors are used for non-sensitive data where error correction isn't necessary, such as streaming audio or video data. I included a function for Form 2 sectors called get_edc_form2, but I'm not sure if I've properly tested it, since I don't modify Form 2 sectors in any of my work. get_edc_form2 takes a 2332-byte string (minus the 16-byte header) and produces a 4-byte EDC.

Actually, the 16-byte header isn't part of the EDC calculation or the ECC calculation, so I guess it doesn't benefit from from error detection at all. When calculating ECC, first you calculate EDC on a 2056-byte data block, then you calculate ECC on the combined 2060-byte data & EDC block.

Clarify?

Also which fields in each sector are used in the calculations?
Find all posts by this user
Quote this message in a reply
09-30-2017, 03:24 PM
Post: #25
RE: PS1 CD-ROM EDC and ECC
Each form 1 sector has the following format:

12 bytes "synchronization pattern"
4 bytes "header"
8 bytes "subheader"
2048 bytes "data"
4 bytes "EDC"
276 bytes "ECC"
Total: 2352 bytes

First, you must calculate EDC on: subheader + data. So EDC covers a total of 2056 bytes.

Then, you must calculate ECC on: four null bytes + subheader + data + EDC. So, the string used in the calculation is 2064 bytes long, including the four null bytes, but the actual data being covered by error correction is only 2060 bytes long.
Find all posts by this user
Quote this message in a reply
10-02-2017, 07:53 AM
Post: #26
RE: PS1 CD-ROM EDC and ECC
I need to figure out how long the data List parameters are for every method, but I just realized that I can do this by adding print statements and loops to your python code so never mind. Consider this a note to self.
Find all posts by this user
Quote this message in a reply
11-17-2017, 04:59 PM
Post: #27
RE: PS1 CD-ROM EDC and ECC
I had to make test programs for both the Python and Java versions, but the crc32 methods in both return the same values. One step done.

Code is attached below for posterity.

edit

More progress on the Java test class. Code attached below. Next part requires translation of some obscure Python 2 string manipulation crap, so it will be another week.


Attached File(s)
.zip  crctest.zip (Size: 114.29 KB / Downloads: 1)
.zip  testClass.zip (Size: 2.45 KB / Downloads: 0)
Find all posts by this user
Quote this message in a reply
12-02-2017, 04:55 PM
Post: #28
RE: PS1 CD-ROM EDC and ECC
Dude what the fuck is going on with method get_edc_ecc(...)? Seriously what the fuck is this shit?

Code:
def get_edc_ecc(data):
    assert len(data) == 0x818
    edc = crc32(data[0x10:0x818]) # 0x818 - 0x10 = 2056 (base 10) aka EDC length
    for _ in xrange(4):#lol why
        data += chr(edc & 0xFF)
        #^wtf is going on here?
        edc >>= 8
    assert len(data) == 0x81c
    assert len(data)-12 == 0x810 #aka ECC length
    temp = encode_L2_P("".join(map(chr, [0, 0, 0, 0])) + data[0x10:])
    #^jesus wtf is this shit?
    temp = encode_L2_Q(temp)
    data += "".join(map(chr, temp[-0x114:]))
    assert len(data) == 0x930
    return data[0x818:0x81c], data[0x81c:]
I fucking hate this fucking language, can you please for fuck's sake provide some goddamn comments?

Why are you converting things back and forth from strings to whatever the fuck they are before the conversion? Why do you do this? What are they before the conversion? What are they after and how long are they?
Find all posts by this user
Quote this message in a reply
12-02-2017, 09:28 PM (This post was last modified: 12-02-2017 09:32 PM by abyssonym.)
Post: #29
RE: PS1 CD-ROM EDC and ECC
Look man, I already linked you to the C version and told you it was probably more like what you're used to. I don't know why you're tormenting yourself with this if you don't even know python. It's not like I'm going to leave comments explaining python keywords and syntax in every bit of code I write.

Code:
for _ in xrange(4):#lol why
It's a damn "for" loop so I don't have to write the code out four times, what are you confused about?

Code:
data += chr(edc & 0xFF)
#^wtf is going on here?
edc >>= 8
Takes the lowest eight bits and converts them to a char and appends that to "data", then bitshifts "edc" by eight bits, obviously.

Code:
temp = encode_L2_P("".join(map(chr, [0, 0, 0, 0])) + data[0x10:])
#^jesus wtf is this shit?
Okay this is a legitimately shitty way to write this. Should've been "\x00\x00\x00\x00". But if you don't know what "map" means or what list slicing is, come on dude, at least educate yourself on the basic features of the language.

Quote:Why are you converting things back and forth from strings to whatever the fuck they are before the conversion?
They're integers. You can only do bitwise operations on integers. Then they're chars after I convert them with "chr" which is a python keyword that converts 8-bit integers to chars. Then they become a string after I append them to "data" because a string of chars is called a string. I understand this is some complicated stuff but I have faith in your ability to figure it out.
Find all posts by this user
Quote this message in a reply
12-03-2017, 03:19 PM
Post: #30
RE: PS1 CD-ROM EDC and ECC
The "lol why" references the use of an underscore as a field name in a for-each loop. I understood what it means (or I did after double checking that your "_" variable isn't reused anywhere else). Why would you do that to yourself when there are perfectly good "i", "j", and "k" to be used like a normal programmer?

Really the only thing that confuses me are the things you need to understand specifically about Python 2 to do this, like how i/o is done with strings for some reason. I don't know what Python 3 is like but the scripting language excreta left over in Python 2 (Bash and I think Perl do all i/o using chars or strings IIRC) makes me wonder why anyone uses it. I know what map is and to be honest I'll be using it once Java 8 reaches my distro (in about 4 years at this rate).

C has it's own set of assumptions about what you know that are in many ways worse than Python 2. That's why I like Java so much; almost everything is out in the open, it favors explicit over implicit. It's Autismâ„¢ as a programming language.

...

In case my laptop is stolen or dies in the next 18 hours, here are my comments thus far.

Code:
def get_edc_ecc(data):
    assert len(data) == 0x818
    edc = crc32(data[0x10:0x818]) # 0x818 - 0x10 = 2056 (base 10) aka EDC length
    for _ in xrange(4):#lol why
        data += chr(edc & 0xFF)
        #edc is a 8 bit unsigned byte? news to me. chr() converts int to string.
        # the += part adds it to the end of the list????
        #maybe print statements to see what's going on
        edc >>= 8 #shift edc 8 bits right...
        #but we already did something with edc???
        #does this affect the field "data"?
        #how does python 2 deal reference primitives?
    assert len(data) == 0x81c
    assert len(data)-12 == 0x810 #aka ECC length
    temp = encode_L2_P("".join(map(chr, [0, 0, 0, 0])) + data[0x10:])
    #^jesus wtf is this shit? join is like string slicing i guess.
    #"" is an empty string - cheap way to make a new string.
    #map(...) should've been "\x00\x00\x00\x00". ~abyssonym
    temp = encode_L2_Q(temp)
    data += "".join(map(chr, temp[-0x114:]))
    #here's the other join(...), the negative index on temp indicates we start...
    #counting from the end of the list.
    assert len(data) == 0x930
    return data[0x818:0x81c], data[0x81c:]
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)