BV Forth supplement 2

Interfacing SD Cards

 

 

 

 

 

 

ByVac

 

 

 

 

 

©ByVac 2007

www.byvac.co.uk

Revision 0.b

 

 

 

Copyright in this work is vested in ByVac and the document is issued in confidence for the purpose only for which it is supplied. It must not be introduced in whole or in part or used for tendering or manufacturing purposes except under an agreement or with the consent in writing of ByVac and then only on condition that this notice is included in any such reproduction.

 

ã        Copyright ByVac 2007

No warranty

THE WORK IS PROVIDED "AS IS," AND COMES WITH ABSOLUTELY NO WARRANTY, EXPRESS OR IMPLIED, TO THE EXTENT PERMITTED BY APPLICABLE LAW, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

 

Disclaimer of liability

IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS WORK, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

 

Revisions

First draft 0.a	December 2007	Applies to BVF V1.206 and above
Second draft 0.b	February 2008	Updated for inclusion in the Forth Library

Introduction

This article will explain how to interface and use an SD (secure digital) card using a FAT16 file system that is compatible with PC’s using Microsoft Windows and Linux. This site contains all of the necessary programs to facilitate this.

Figure 1 BV511 connected to 1G SD Card

The software presented is by no means complete but does provide a workable system that can be used for example for data logging. The big advantage is that any files created by the application can be read by a PC at a later date.

The user should be aware that this code has not been fully tested and there is a possibility of damaging the SD card, or at least the data on it so it is suggested that an old card be used at first. Also no checking is done that the card has been formatted to FAT16. This software only works with FAT16 cards, usually in the range 16mB to 2Gb. The code will happily write to a FAT12 or FAT32 card but it will not be readable by a PC later.

Resources

This article along with all of the software can be found in the downloads section of www.pin1.org A suitable SD card holder can be obtained form www.byvac.com. All of the Forth software for this article is on this site accessible by the menu on the left.

Note that the files on this site vary slightly form the files that can be downloaded form pin1.org but essentially they do the same thing. This site will be more up to date then the pin1 files.

The SD Card

The Secure digital card has two modes of operation, SD mode and SPI mode. The former is designed for high speed purpose built applications such as cameras and MD3 players. The second SPI mode is designed for as of interface and general purpose use with microntrollers. It is a bit slower but uses the built in SPI interface of the BV511 so it is this mode that will be used.

The use of MMC (Multi Media Card) is often used when referring to this topic, that is because they are directly interchangeable at this level. The SD card is the next generation on from the MMC. Although not tested is quite likely that an MMC card will work with the same software.

Hardware Interface

The SD card has a 9 pin interface that i suppose could be soldered directly to the BV511, however a holder is much more convenient.

Pin	SPI Mode
	Name	Type	Description
1	CS	I	Chip selection in low status
2	MOSI	I	Data in
3	VSS	S	Ground
4	VDD	S	Power supply 3.3V
5	CLK	I	Clock
6	VSS2	S	Ground
7	MISO	O/P	Data output
8	n/c
9	n/c

Table 1 pin description

The above table shows the pin designations for an SD card. Normally pins 4 and 6 are connected together and note that this is a 3V logic device not a 5V one.

Six wires, that include the power supply are required for connection to the BV 511. The MISO (Master In Slave Out), the MOSI (Master Out Slave In) and CLK pins must be connected as shown. The CS (Chip Select) can be any pin, but using p0.3 will match the software without any modification.

The above represents the minimal configuration. Two extra ports could be used for Card detect and write protect if required. NOTE again that SD cards work on 3v3 not 5V.

Software

The software at first site is fairly extensive this is because, using the whole suit, it is capable of reading and writing using the FAT filing system with just a few limitations. It is not all required however for simpler applications. It can be broken down as follows:

·         Basic reading and writing of the SD card using memory addresses. An actual physical address is specified for R/W to the card

·         Reading and writing to the card using 512 byte sectors including specifying a file number and a card so that multiple cards can be used. There is no file system so it will not be compatible with a PC but is very easy to implement a simple mass data storage.

·         A filing system capable of reading and writing to a FAT16 formatted card using file names and logical sectors with random read and random write. Files can be transferred to and from a PC.

To get the full functionality requires some fairly extensive code, if this is not required then the second option gives very easy reading and writing to and from the card with very little effort. The next part presents the three levels of complexity.

Part 1

This section describes the raw reading and writing to the SD card and used the following files:

·         Spi.flb

·         MMC.fth

The first file is a small interface to the SPI hardware, it contains the constants and initialisation for all of the SPI hardware. The main routine from this file is a word called SPI. A recap on how this works is that it takes a byte in and at the same time puts one out so if you want to read something then you need to provide a dummy byte, usually 0xFF. By the same token if you need to write something then the byte that comes out is often dropped.

The MMC file is the basic interface for the MMC card (SD Card). It provides sufficient functionality to be able to read from and write to the card using a block format. A block format is, in this case 512 bytes at a time.

Provided the basic two words, mmc.read and mmc.write remain the same then this file can be altered to use the faster SD type interface rather than the SPI interface. There will be no need to change any of the rest of the software to use it. There is also no CRC error checking that may be incorporated if required.

Using the above two files may be sufficient for some purposes. The SD card can be used as an extension of memory, ideal for a simple data logger and much more capacity than a serial EEPROM.

Using Stage 1

To use the SD card at this level use “stage-1.fth” this will take care of the scope ID and only expose those words that are needed for this level of use. Incorporated into this file is “dump.fth” which is useful for looking at the contents, it can be discarded for real applications.

To start the card use 3 MMC.START, this will return a value greater than 0 to indicate that the card has been initialised correctly, a value of 0 means that it hasn’t, so check the card and connections. The ’3’ is the pin used for Chip Select, more than one card can be connected to the system at any one time by using different chip selects. The word START in the file does this and checks the return value.

Once the card has been started successfully then you can read and write to any physical address on the card. *** Warning if this has been formatted to FAT then depending on where you write may spoil the format and the card may need to be re-formatted for use with a PC.

The “stage-1.fth” gives an example of reading from an address on the card after initialising it. The SPI interface has a very low default speed so this is quickened up a bit, 400kHz will do all cards and most can go faster.

Using just these few files will now allow the card to be used as a very large linear memory. If that is all that is needed then go no further, however you may want to consider the next part as this introduces sectors that improve this process.

Part 2

·         fat-1-a.flb

The above extends the basic functionality to enable reading and writing of numbered sectors rather than memory addresses, this is much easier to manage when the data is organised in blocks. It also provides a buffer from which to read and write and in addition to that it has the beginnings of the FAT system because it is this file that reads the card FAT geometry. In summary the file has the following functions.

·         Provides two buffers, one for the system and one for the data

·         Associates a device to a system buffer and a file to a data buffer

·         Provides words for reading and writing 16 bit words, Intel format

·         Words for reading from and writing to sectors rather than memory addresses

·         Reads the boot block and sets the variables that define the disk geometry for a FAT filing system.

At this stage some limited compatibility with a PC can be had by reading and writing to known sectors for a particular file. It is more likely to be used though for internal storage. As an example a temperature logger could be built that writes the sectors on the card in turn. When reading out the information this could be sent to the USB on the 511 and read by a terminal type program.

Using Stage 2

The fat-1-a library exposes the following words:

mmc.mount ( cs device -- ) // used for initialising the card
fb ( -- buffer-address ) // for access to the data buffer
geo. ( -- ) // prints out the geometry
sec@ ( sector -- ) // fetch a sector into data buffer
sec! ( sector -- ) //  write data buffer to supplied sector

These are the basic words but you may want to expose more of the geometry words, you may need to know the FAT type for example. This would be done by making public access to the device array.

As an example, load the stage-2.fth file and type start ( assuming the hardware is connected), then type gx, you should get something like the following:

The screenshot shows a card that I had in the holder, a 16MB card formatted using FAT16. The size of the card can be seen at the bottom, 14 MB, some space is lost due to the formatting. This information is stored in an array as it is being used all of the time. Try:

59 disp

On the card in the above screenshot this will get sector 59 into the file buffer (fb) which is the root directory and then dump this to the screen, on your system use whatever number is next to the ‘Dir’ entry.

This is the screen shot from using 59 DISP, the sector is loaded into memory and displayed, you can see how the directory is formatted.

Using ‘sec@’ and ‘sec!’ greatly simplifies reading and writing to the card, but again it can only be used for internal systems, for PC compatibility the FAT system must be maintained. This is done in part 3.

Part 3

This next part offers full reading and writing using the FAT16 filing system. It takes a reasonable amount of code to achieve this, hence part 1 & 2 above. If using a Windows operating system it has been found that a 16MB card is formatted to FAT12 and there is no option to do otherwise. I have used Ubuntu with some instructions found on the web to format theses cards to FAT16. Cards greater than 16MB, well 1G cards are formatted to FAT16. There is an upper limit of 2GB for FAT16, cards greater than this must either be partitioned or use FAT32. These other FAT systems are out of scope but there is nothing to prevent the user from modifying the code to read and write these as well. FAT32 is an easy modification but FAT12 has a very awkward FAT table format.

The following additional three files are used in this section:

·         Fat-2.fth

·         Fat-3.fth

·         Fat-4.fth

Fat-2: This mainly takes care of directory listing and reading the directory, there are some necessary words in here for also writing to the directory but this is taken care of by the next files.

Fat-3: This is for reading files and introduced the concept of a logical sector. This is a sector WITHIN the file rather than on the card. Sector 0 is always the first sector of that file but it could actually be sector 637 on the card. It was not possible to introduce this concept until the file, ‘file open’ had been established. Also contained in this file is the beginning of a method to write to the FAT itself using cluster numbers. As it turns out writing to a cluster is similar to reading and writing a Forth variable, where the address @ and ! are used. The difference is that word16 and word16! are used.

One other complication when using clusters is that the NEXT cluster address is given by reading the CURRENT cluster. When updating the FAT table it is usually necessary to keep track of the current cluster as well as the next.

In a larger system the FAT and probably the directory would be kept in RAM to speed up this process. In this system there is not enough room for that, the sacrifice for this is speed as each time a write is made it must be written back to the card.

Fat-4: This contains the words used for writing to a file. The main word to come out of the collection is a random write. One of the more complex aspects of this is when the file is not currently large enough to accommodate the next write. As an example suppose a file is 1 cluster big (4 sectors 2k) and a random write to logical sector 35 is requested. The file has to be expanded to 9 clusters to accommodate this. This involves updating the FAT Table and the directory entry to reflect the new size.

Using Stage 3

If all of the files so far were to be included they would not all fit into RAM and so an intermediate save would be required, however when using the library feature of BVT is not necessary for the following examples. If you were developing this system than it may be wise to save most of the files you need to make loading quicker.

Fat-2

Fat 2 extends the functionality to give directory access. It has words for listing the directory and finding files within the directory. When find is used it populates the system buffer dbuff and the associated arrays with the information for that file so this is an important step for using files.

f.device ( device -- ) // set device

f.find ( name -- [array] ad -1!0 ) // find file given string address

f.ls ( -- )   // dir of current device

 

f.device is a number, this points to the correct device array that was used when mounting the card, there is only one device implemented in the default software.

f.find will search the directory for that entry, it expects a string to be at the address given and the file to be in ‘dotted’ format i.e. fred.txt. If an entry is found it will return -1 and the directory entry number. This number corresponds to the position that entry is in the directory, 0 being the first entry. To calculate the actual offset into the sector multiply by 32.

f.ls This will list the root directory, it is similar to LS in Linux and DIR in windows.

Stage-3-a.fth

This file will load all of the necessary libraries and give the functionality to be able to mount the device (start), list its geometry (gx), display a physical sector (disp) and list the directory contents (dir). This is only a demonstration, there is much more functionality in the files but these are the main words that may be required.

An important note, that from now on we are extending fat-1-a.flb, because fat-2,3 and 4 use so many of each others words it would not be practical, or sensible to make them all public. To this end make the SID the same for all of these files - you can see this by looking at the head of Stage-3-a.fth.

Here shown is the directory listing of the card shown earlier, f.ls reads and formats the sector.

Fat-3

This file contains the required words for reading files and introduces the concept of a logical sector, this is a sector WITHIN the file rather than on the card. There is no need to close a file when reading. The random read provided is the most flexible, a serial file read can be implemented by the user either by using the file size or the -1 that is returned when  the end of file is encountered.

f.open ( string handle -- -1|0) // opens file

f.rrandom ( handle sector -- 0|-1) // -1 on past end of file

 

f.open requires the address of a sting containing a file name and a file handle. The file handle is a number from 0 to however many file buffers have been defined in fat-1-a. This number simply tells the system to use that particular array and thus more then one file can be opened at the same time. Note that the card must be mounted first. On open success a -1 is returned.

f.rrandom This is read random, the specified sector is NOT the absolute sector but a sector within the file, so the start of the file will always be sector 0. The handle number is the same number that was used to open the file. NOTE 0 is returned if the file has been read okay, -1 is returned if an attempt is made to read past the cluster. This is not the end of the file but the limit of what the file is allowed to use. The file size should be used to determine the end of the file.

Stage-3-b.fth

Stage 3 is a demonstration of of opening two files and reading from them. To make this happen fat-1.flb has already had "fileBuffers" set to 2 so there is room. To use the words contained within this file needs two text files called fred-1.txt and fred-4.txt. You can download the test file here, unzip it and use the PC to put it onto a card.

Don't forget to use 'start' to initialise the card first. One of the first words s1a, simply shows that the file buffer contains the current directory entry. It is useful to keep it here so it can be updated when writing to the file. This is what an open file that has been written to must be closed otherwise this information does not get updated.

To open an existing file, supply the address of a sting and a file handle to the f.open word as shown. Open should return -1 on success, so this code will abort if a 0 is returned.

Both of the files can now be read using their handle:

The file is read by using f.rrandom, in the example 0 0 f.rrandom will read the first sector of ‘fred-1.txt’ as this has the associated file handle 0. f.rrandom should return 0 on success. When f.rrandom is used it gets the specified sector from the card and places it in the file buffer associated with the handle. Each handle has its own 512 byte file buffer. To see the contents of the file buffer f.fb is used. In the first instance 0 f.fb is used to get the buffer associated with handle 0 and in the second instance 1 f.fb is used to get the second file buffer.

Fat-4

The words in this file are used for writing to the card, updating the directory and FAT tables. It is the most complex of all of the files. Because files can be created and updated then a real time clock is required, there is one built into the BV511 and the library for this is rtc.flb

The public words are as follows:

f.create ( string handle -- -1|0 )  // creates and opens file

f.reset ( string handle -- )  // open and new

f.del ( string -- )  // deletes file

f.close ( handle -- )  // closes, updates date + size

f.wrandom ( handle sector -- )  // write random

 

f.create This will create and open a file using the file name at the string address and the handle given, a -1 is returned on success. The file name must be 8 characters or less with a 3 character or less extension ( old DOS format), no checking is done do beware. This creates an empty file ready to be written to.

f.reset This will open an existing file and erase all of its contents.

f.del This will delete a directory entry and free up card space.

f.close This is not required for reading the card but if the card has been written to a local copy of the directory entry is kept in RAM. If the file is not closed this RAM is not copied back to the card and so, although the data may have been written to the card it may not be accessible because the directory has not been updated. This word shoud be used to close a file if it has been written to.

f.wrandom will take the contents of the file buffer (pointed to by the given handle)  and write it to the specified sector on the card. Room is made in the file to accommodate whatever sector is specified, within the limits of the card, but this is not checked. The sector is the logical sector within the file, sector 0 is at the beginning of the file.

Using Stage 4

Stage-4.fth

This will not fit into 16k of RAM in one go so you need to load stage-3b first and save it. Then load this on top.

The code in this file is explained as follows:

Create a file and write to it

Use start first to mount the card and possibly DIR so that you can verify the file has been created. To create a file the address of a string and file handle must be supplied.

The above will create a file called “joe.txt” and associate this with file handle 0. f.create will return if successful. If the file already exists it will return 0, using this mechanism it is easy to create a file if it does not exist otherwise open it.

Once created the file can be read and written to. To write to a file f.wrandom is used. This will write a 512 byte block of RAM to the given logical sector on the card. If the sector is outside the current file range the file will be expanded to accommodate it.

In this example a string is moved into the buffer of handle 0 using the MOVE word. It is then written to the specified sector. Two writes are made to this file one to sector 0 and the other to sector 57. The fill will expand to about 32k to accommodate the write to sector 57. To make sure that the directory entry is updated a close command terminates the S3 word.

As a further example, this opens up and existing file and uses a loop to write to 67 sectors. The card can be pulled, put into a PC card reader and observed using notepad. CLR0 simply sets the sector to all 00 so that it looks neater.

Take a look at the contents of joe.txt using a text editor on a PC, each sector begins with the word 'sector' and has the sector number following it. There are no CR so if word wrap is on it will not look very neat. S9 makes some attempt to make the text look neater.

Limitations

There are several limitations to this system, these can be improved by modifying this software, please share this if you do.

·         The MMC card uses the SPI interface, it can be made much faster by using the 4 wire SD interface.

·         Only works with FAT16, this can easily be changed.

·         It has not been tested with multiple devices

·         Only one FAT table is updated

·         Old DOS filenames used

·         Very little error checking

 

The File Handle and Operation

This suit of programs took many iterations to get this end result. There are so many choices and decision to be made along the way that will effect something much later down the road. The most significant items are the two arrays, one holds the device information and the other holds the file information.

These arrays are established in fat-1 and are created dynamically as the file is loaded. To make access easier there are two important constants d# for the device and f# for the file. These control which device or file is being referred to, setting f# to 0 will refer to handle 0 and f# to 1 will set the file handle to 1.

The device array holds all of the information about the particular device, size, bytes per sector etc. To use the array D is used, as an example to get the sector where FAT Table 2 is stored use: FATTable2 d @. Also F is used in a similar way for accessing the file array.

File Handle

It became clear that when working with files, information about the file needs to be kept in RAM as this is being accessed and updated all of the time. The file handle is stored in an array and the index of the array (set by the constant filebuffers ) is the number of handles available.

The structure of the handle is as follows:

Offset into array	Name	Description
0	EOF	This may now be obsolete but it was originally for keeping track of the end of the file
4	Device	This associates the device array to this file. This aspect of the code has not been tested
8	IsOpen	Set to -1 when this handle is in use, f.close sets this back to 0
12	DirSec	This is the physical sector where the directory entry is stored. Not sure if this is needed
16	En#	This is the entry number in the directory for this file. All files have an entry number 0 being the volume name of the card.
20	dirEntry	This is the actual directory entry for the file in its raw state, it is 32 bytes big.
52	Ftabb	This is the start of a 512 byte file buffer that is accessed with f.fb. This area of RAM is filled by read and it is put back to the card using write

 

The file handle therefore also contains a data buffer so the total size is 564 bytes per handle. Previous incarnations of this software had only one file buffer and several smaller handles. This is workable but less efficient in terms of code size and speed.