Unix detecting file encoding

Just look at the man page. Or, failing that, use file -i linux or file -I osx. That will output MIME-type information for the file, which will also include the character-set encoding. I found a man-page for it, too :. It is really hard to determine if it is iso If you have a text with only 7 bit characters that could also be iso but you don't know. If you have 8 bit characters then the upper region characters exist in order encodings as well. Therefor you would have to use a dictionary to get a better guess which word it is and determine from there which letter it must be.

But if a file is part of a repetitive process then the file might have different content in the future that could invalidate the guess. Show 2 more comments. Active Oldest Votes. To some extent, ewcz's advice works. The huge advantage of uchardet is that it analyses the whole file just tried with a 20GiB file as opposed to file and enca — tuxayo. Add a comment. Falaen Falaen 2 2 silver badges 12 12 bronze badges.

Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. But yes, really, you should know! Often, 'C' is linked to iso or Latin-1, but your file is not that.

Sorry DGPickett, tried that and it looked all Greek to me not in a literal sense, lol. Well, utf-8 and unicode have a pattern in their encoding. The dd command has an ebcdic decoder I have used.

Might it be from big blue land? There is a 'chardet' python based tool. Originally Posted by DGPickett. Yes, IBM is a world unto itself, and ebcdic is the dominant charset, and even then to print right you may need the code page. The r-x-0 rows of the card became upper bits, and were binary coded. You can probably get enca binary or source, and python and chardet for free, and install them.

Join Date: Jul Did you consider using iconv or recode? Maybe on a trial and error basis, but I think they complain if an unsuitable from-charset is given.

Shell Programming and Scripting. How to know file encoding? There's no need to parse file output, file -b --mime-encoding outputs just the charset encoding — jesjimher. Add a comment. I'm not delighted about yet more packages, yet sudo apt-get install uchardet is so easy that I decided not to worry about it As I just said in a comment above: uchardet falsely tells me the encoding of a file was "windows", although I explicitly saved that file as UTF However, encguess guess correctly, and it was pre-installed in Ubuntu Excellent, works perfectly.

Mohiuddin Ahmed. For your question, you need to use mv instead of iconv :! Wolfgang Fahl Wolfgang Fahl As pointed out on MacOS this won't work: file -b --mime-encoding Usage: file [-bchikLNnprsvz0] [-e test] [-f namefile] [-F separator] [-m magicfiles] [-M magicfiles] file Encoding is one of the hardest things to do, because you never know if nothing is telling you.

Norbert Hartl Norbert Hartl 9, 5 5 gold badges 34 34 silver badges 46 46 bronze badges. It may help to try to brute force. Then one would need to manually check the output searching for a clue into the right encoding. Of course, you can change the filtered formats replacing ISO or WIN for something appropriate or remove the filter by removing the grep command.

In PHP you can check it like below: Specifying the encoding list explicitly: php -r "echo 'probably : '. Mohamed23gharbi Mohamed23gharbi 1, 22 22 silver badges 27 27 bronze badges. I'm not talking about literal translation such as: English French of de, du and et the le, la, les although that's possible. If you really, really care about the encoding you need to validate it yourself. Matyas Matyas You can extract encoding of a single file with the file command.

I have a sample. Daniel Faure Daniel Faure 4 4 silver badges 12 12 bronze badges. I made this script to convert all to utf! Teocci Teocci 4, 1 1 gold badge 39 39 silver badges 40 40 bronze badges. With Perl, use Encode::Detect.

Can you give an example how to use it in the shell?