Advice on using a header in text to re-organize a large dataset

I made a couple of assumptions to generate a way to split the files by genes
Assumptions:

• You are working with plain .txt files
• You want the new organized genes in plain .txt files (there are a million other things you may want but I figured this was the simplest)

Rather than doing this in bash bc that is far from my strong suit I wrote a simple ruby script that takes the raw files from one folder and parses them by Gene and writes the new files to a new folder. With this you only end up with 1 file for each Gene.

path = 'genes'
Dir.foreach(path) do |filename|
  next if filename == '.' || filename == '..' || filename == '.DS_Store'
  gene_file = nil
  header = true
  puts "working on #{filename}"

  file = File.open("#{path}/#{filename}", 'r')
  file.each_line do |line|
    puts line
    if line.empty? || line == "\n"
      puts "line empty"
      header = true
      gene_file&.close
    elsif header
      puts "opening new file"
      gene_file = File.open("new_gene_files/#{line.gsub(/_id\d/, '').gsub('>', '').strip}.txt", 'a')
      puts "adding to file"
      gene_file.puts line
      header = false
    else
      puts "adding to file"
      gene_file&.puts line
    end
  end

  file.close
end

From what you wrote, I assume that you are looking for an intermediate data structure to be able to further process your data, is that correct?

The second question is: do you strictly want to stick to plain files?

And finally, is searching or retrieving of files an issue for you? (Is it enough to know the gene and the id if a file or do you want to be able to list all ids for gene x?

If you want to stick to one file per gene and id and the number of files inside one folder is an issue, you could create subfolders, either by gene name or if that's still too much, do a breakdown by the first letter of the gene name first, then have a folder with the gene name and finally the ids in there.

If you want to be more flexible with what you do with your data, you can parse the data into a csv file or sqlite database. There, you can have one column for the gene, one for the id and one for the actual generic information. Then you can iterate over the lines (which is a bit more comfortable with sqlite) depending on your desired analysis.

If I got something wrong or you need to achieve something more with your data, I am happy to hear about it.

FYI, this is my sister 🙌.

Hope you get some helpful replies, Hannah!

What's the file format of each gene file? If it's something like a .csv you could probably run each file through a script that detects the header line, and then add it to an array. Once you've ran through all your files, you would have that array and then write a new file like all_gene1.csv or something.

Since you have key-value pairs already with the header and the genetic data, you already have a good data structure that you can work off of. I think Ruby can handle this pretty well, but I'd have to play with the data in order to figure out if it's really possible, and you probably can't hand over the data so easily. 🙃

Edit: actually, the file format probably doesn't matter for Ruby. I think you can read line by line in Ruby regardless of file format.

Thank you all! It's in a .fasta file which is basically a plain text file. This is all really interesting -- I'll definitely look into sqlite and how to convert the data into a CSV so it's easier to work with. Also seems like a good time to brush up on my python! :)