Microsoft Visual FoxPro / dBase Library for Go

A comprehensive Golang package for reading, writing, and managing FoxPro dBase table and memo files.

Overview

This package provides comprehensive tools for working with dBase-format database files (.DBF) and their associated memo files (.FPT). It offers cross-platform compatibility with optimized I/O operations for both Unix and Windows systems, flexible data representation, and safe concurrent operations.

Key Features

📁 Full dBase Support: Read and write .DBF tables and .FPT memo files
🔄 Multiple File Versions: Support for FoxPro, FoxBase, and dBase III/IV formats
🌐 Encoding Support: 13+ character encodings with automatic code page detection
🔒 Concurrent Safe: Built-in synchronization for multi-threaded applications
📊 Flexible Output: Convert to Go structs, JSON, maps, or native types
🔍 Advanced Features: Search, navigation, exclusive file access, and table creation
⚡ Memory Efficient: Streaming reads without loading entire files into memory
🛠️ Developer Friendly: Comprehensive error handling and debugging support

Use Cases

Legacy System Migration: Modernize applications that rely on dBase files
Data Integration: Import/export data from legacy business systems
File Conversion: Convert dBase files to modern database formats
Backup & Recovery: Create tools for dBase file manipulation and repair
Business Intelligence: Extract data from legacy ERP/CRM systems

Quick Start

Installation

go get github.com/valentin-kaiser/go-dbase@latest

Basic Usage

package main

import (
    "fmt"
    "log"

    "github.com/valentin-kaiser/go-dbase/dbase"
)

func main() {
    // Open a dBase file - you must specify exactly one data source
    config := &dbase.Config{
        Filename: "example.dbf", // Filesystem access
        // Data: []byte("..."),  // OR byte data  
        // Reader: someReader,   // OR custom reader
        // IO: customIO,         // OR custom IO implementation
    }
    
    table, err := dbase.OpenTable(config)
    if err != nil {
        log.Fatal(err)
    }
    defer table.Close()
    
    // Read all rows
    for !table.EOF() {
        row, err := table.Next()
        if err != nil {
            log.Fatal(err)
        }
        
        // Access data by column name
        name, _ := row.StringValueByName("NAME")
        age, _ := row.IntValueByName("AGE")
        
        fmt.Printf("Name: %s, Age: %d\n", name, age)
    }
    
    // Convert row to map
    row, _ := table.Row()
    dataMap, _ := row.ToMap()
    fmt.Printf("Row as map: %+v\n", dataMap)
    
    // Convert row to JSON
    jsonData, _ := row.ToJSON()
    fmt.Printf("Row as JSON: %s\n", jsonData)
}

Features Comparison

Comparison with other popular Go dBase libraries:

Feature	go-dbase	go-dbf	go-foxpro-dbf
File Operations
Read .DBF files	✅	✅	✅
Write .DBF files	✅	✅	❌
Read .FPT memo files	✅	❌	✅
Write .FPT memo files	✅	❌	❌
Data Features
Full data type support	✅	❌	❌
Character encoding support	✅ (13+ encodings)	✅*	✅ (extensible)
Automatic code page detection	✅	❌	❌
Data Conversion
Go struct mapping	✅	❌	✅
JSON conversion	✅	❌	✅
Map conversion	✅	❌	✅
Advanced Features
Concurrent access safety	✅	❌	❌
Exclusive file locking	✅	❌	❌
Search functionality	✅	❌	❌
Table creation	✅	❌	❌
Database operations	✅	❌	❌
Memory efficiency	✅	❌	✅

Technical Advantages

🔒 Concurrent Safety: Built-in mutex locks ensure safe operations in multi-threaded environments.

⚡ Memory Efficiency: Streaming approach reads only required file positions instead of loading entire files into memory, enabling processing of large files with minimal RAM usage.

🔐 Exclusive Access: Support for exclusive file locking during write operations prevents data corruption from concurrent access.

🌐 Encoding Intelligence: Automatic code page detection and conversion with support for 13+ character encodings.

🛠️ Developer Experience: Comprehensive error handling with detailed trace information and extensive helper methods.

Supported Data Types

All column values are returned as interface{} with helper methods for type-safe conversion.

dBase Type	Type Name	Go Type	Description
Text Data
C	Character	`string`	Fixed-length text fields
M	Memo	`string`	Variable-length text in .FPT file
V	Varchar	`string`	Variable-length text
Numeric Data
N	Numeric (no decimals)	`int64`	Integer numbers
N	Numeric (with decimals)	`float64`	Decimal numbers
I	Integer	`int32`	32-bit signed integers
F	Float	`float64`	Floating-point numbers
Y	Currency	`float64`	Currency/money values
B	Double	`float64`	Double-precision floats
Date/Time Data
D	Date	`time.Time`	Date values (YYYYMMDD)
T	DateTime	`time.Time`	Date and time values
Other Data
L	Logical	`bool`	Boolean values (T/F)
W	Blob	`[]byte`	Binary large objects
Q	Varbinary	`[]byte`	Variable-length binary
G	General	`[]byte`	General/OLE objects
P	Picture	`[]byte`	Picture/image data

Type Conversion Examples

// Type-safe value retrieval
name, err := row.StringValueByName("NAME")        // string
age, err := row.IntValueByName("AGE")             // int64  
salary, err := row.FloatValueByName("SALARY")     // float64
active, err := row.BoolValueByName("ACTIVE")      // bool
hired, err := row.TimeValueByName("HIRE_DATE")    // time.Time
photo, err := row.BytesValueByName("PHOTO")       // []byte

// Panic versions (for when you're sure the field exists)
name := row.MustStringValueByName("NAME")
age := row.MustIntValueByName("AGE")

📖 Reference: Microsoft Visual Studio FoxPro Data Types

Note: Need additional column types? Please open an issue or submit a pull request.

Character Encoding Support

Automatic detection and conversion of 13+ character encodings with UTF-8 as the standard:

Code Page	Platform	Identifier	Description
437	U.S. MS-DOS	`0x01`	Original IBM PC character set
850	International MS-DOS	`0x02`	Western European
852	Eastern European MS-DOS	`0x64`	Central/Eastern European
865	Nordic MS-DOS	`0x66`	Nordic countries
866	Russian MS-DOS	`0x65`	Cyrillic script
874	Thai Windows	`0x7C`	Thai script
1250	Central European Windows	`0xC8`	Central European
1251	Russian Windows	`0xC9`	Cyrillic Windows
1252	Windows ANSI	`0x03`	Western European Windows
1253	Greek Windows	`0xCB`	Greek script
1254	Turkish Windows	`0xCA`	Turkish script
1255	Hebrew Windows	`0x7D`	Hebrew script
1256	Arabic Windows	`0x7E`	Arabic script

Encoding Examples

// Automatic encoding detection (recommended)
config := &dbase.Config{
    Filename:          "data.dbf",
    InterpretCodePage: true,  // Auto-detect from file
}

// Manual encoding specification
config := &dbase.Config{
    Filename:  "data.dbf", 
    Converter: dbase.ConverterFromCodePage(0x03), // Windows-1252
}

// Custom encoding registration
import "golang.org/x/text/encoding/charmap"
dbase.RegisterCustomEncoding(0x99, charmap.ISO8859_15)

All encodings are automatically converted to/from UTF-8 for seamless Go integration.

Data Source Configuration

The library supports four different data sources, and you must specify exactly one when opening a table:

1. Filesystem Access (Filename)

config := &dbase.Config{
    Filename: "data.dbf",
    // Optionally specify other settings
    TrimSpaces: true,
}

2. Byte Data (Data)

// Read file into memory
dbfData, _ := os.ReadFile("data.dbf")
memoData, _ := os.ReadFile("data.fpt") // Optional memo file

config := &dbase.Config{
    Data:     dbfData,
    MemoData: memoData, // Optional
}

3. Reader Interface (Reader)

// Use file handles
dbfFile, _ := os.OpenFile("data.dbf", os.O_RDWR, 0644)
memoFile, _ := os.OpenFile("data.fpt", os.O_RDWR, 0644) // Optional

config := &dbase.Config{
    Reader:     dbfFile,
    MemoReader: memoFile, // Optional
}

// Or use custom readers
dbfReader := dbase.NewBytesReadWriteSeeker(dbfData)
config := &dbase.Config{
    Reader: dbfReader,
}

4. Custom IO (IO)

config := &dbase.Config{
    IO: customIOImplementation, // Implement the dbase.IO interface
}

Validation

The library validates that exactly one data source is provided:

// ❌ ERROR: Multiple data sources
config := &dbase.Config{
    Filename: "data.dbf",
    Data:     []byte("..."),  // This will cause an error
}

// ❌ ERROR: No data source 
config := &dbase.Config{
    TrimSpaces: true, // Only settings, no data source
}

// ✅ VALID: Exactly one data source
config := &dbase.Config{
    Filename: "data.dbf",
}

Advanced Examples

Creating a New Table

// Define columns
columns := []*dbase.Column{
    dbase.NewColumn("ID", dbase.Integer, 0, 0, false),
    dbase.NewColumn("NAME", dbase.Character, 50, 0, false),
    dbase.NewColumn("SALARY", dbase.Numeric, 10, 2, false),
    dbase.NewColumn("ACTIVE", dbase.Logical, 0, 0, false),
}

config := &dbase.Config{Filename: "employees.dbf"}
table, err := dbase.NewTable(dbase.FoxPro, config, columns, 0, nil)
if err != nil {
    log.Fatal(err)
}
defer table.Close()

// Create and add a new row
row := table.NewRow()
row.SetValueByName("ID", 1)
row.SetValueByName("NAME", "John Doe")
row.SetValueByName("SALARY", 75000.50)
row.SetValueByName("ACTIVE", true)

err = table.AppendRow(row)
if err != nil {
    log.Fatal(err)
}

Working with Memo Fields

// Reading memo data
memoText, err := row.StringValueByName("DESCRIPTION")  // Text memo
memoBytes, err := row.BytesValueByName("DOCUMENT")     // Binary memo

// Writing memo data
row := table.NewRow()
row.SetValueByName("DESCRIPTION", "Long text content...")
row.SetValueByName("DOCUMENT", []byte{0x89, 0x50, 0x4E, 0x47}) // Binary data

Search and Navigation

// Search for specific values
field := &dbase.Field{Name: "STATUS", Value: "ACTIVE"}
results, err := table.Search(field, true) // exactMatch = true

// Navigation
table.GoTo(10)           // Go to specific record
table.Skip(5)            // Skip 5 records forward
table.Skip(-3)           // Skip 3 records backward

// Check position
if table.EOF() {
    fmt.Println("At end of file")
}
if table.BOF() {
    fmt.Println("Before first record")
}

Convert to Different Formats

// Convert row to Go struct
type Employee struct {
    ID     int32   `dbase:"ID"`
    Name   string  `dbase:"NAME"`
    Salary float64 `dbase:"SALARY"`
    Active bool    `dbase:"ACTIVE"`
}

var emp Employee
err := row.ToStruct(&emp)

// Convert to JSON
jsonBytes, err := row.ToJSON()

// Convert to map
dataMap := row.ToMap()

Error Handling and Debugging

// Enable debug mode
dbase.Debug(true, os.Stdout)

// Comprehensive error handling
if err != nil {
    if dbaseErr, ok := err.(dbase.Error); ok {
        fmt.Printf("dBase error: %s\n", dbaseErr.Error())
        // Access error details and trace
    }
}

Working with Large Files

config := &dbase.Config{
    Filename:    "large_file.dbf",
    ReadOnly:    true,        // Read-only for better performance
    TrimSpaces:  true,        // Automatically trim string values
}

table, err := dbase.OpenTable(config)
if err != nil {
    log.Fatal(err)
}
defer table.Close()

// Process in batches
batchSize := 1000
processed := 0

for !table.EOF() {
    batch := make([]*dbase.Row, 0, batchSize)
    
    // Collect batch
    for i := 0; i < batchSize && !table.EOF(); i++ {
        row, err := table.Next()
        if err != nil {
            log.Printf("Error reading row %d: %v", processed+i, err)
            continue
        }
        batch = append(batch, row)
    }
    
    // Process batch
    processBatch(batch)
    processed += len(batch)
    fmt.Printf("Processed %d records\n", processed)
}

Complete Examples

Explore comprehensive examples in the examples directory:

📖 Reading Files - Basic file reading and data access
✏️ Writing Files - Creating and modifying dBase files
🔍 Search Operations - Finding specific records
🆕 Table Creation - Building new tables from scratch
📊 Database Export - Converting to modern formats
📋 Documentation - Generating table documentation
🗂️ Schema Analysis - Analyzing table structures
💾 Bytes Data Source - Working with in-memory byte data
🔌 Reader Data Source - Using custom io.ReadWriteSeeker implementations
🔧 Custom IO - Advanced custom data source integration

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

This project is licensed under the BSD 3-Clause License - see the LICENSE file for details.

Disclaimer

⚠️ Important: This library is designed for working with existing dBase files and legacy system integration. While it supports creating new tables, it should not be used to develop new applications that rely on dBase as the primary database format. Consider modern database solutions for new projects.

Name		Name	Last commit message	Last commit date
Latest commit History 767 Commits
.github		.github
dbase		dbase
examples		examples
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
go-dbase.png		go-dbase.png
go.mod		go.mod
go.sum		go.sum

License

valentin-kaiser/go-dbase

Folders and files

Latest commit

History

Repository files navigation

Microsoft Visual FoxPro / dBase Library for Go

Overview

Key Features

Use Cases

Quick Start

Installation

Basic Usage

Features Comparison

Technical Advantages

Supported Data Types

Type Conversion Examples

Character Encoding Support

Encoding Examples

Data Source Configuration

1. Filesystem Access (Filename)

2. Byte Data (Data)

3. Reader Interface (Reader)

4. Custom IO (IO)

Validation

Advanced Examples

Creating a New Table

Working with Memo Fields

Search and Navigation

Convert to Different Formats

Error Handling and Debugging

Working with Large Files

Complete Examples

Contributing

License

Disclaimer

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 71

Uh oh!

Contributors 9

Uh oh!

Languages