initial commit via gitinitshit
This commit is contained in:
+12
@@ -0,0 +1,12 @@
|
||||
# da soft link to latest hibp multi file dir
|
||||
hibp_list
|
||||
# dirs
|
||||
logs
|
||||
normalized
|
||||
backup_files
|
||||
lists
|
||||
# temp, test, backup, and log files
|
||||
*.tmp.txt
|
||||
test_*
|
||||
*.bak
|
||||
*.log
|
||||
@@ -0,0 +1,56 @@
|
||||
# Notes
|
||||
`hibp_sha1_dir` is a softlink to the latest haveibeenpwned sha1 multi-file dir `ln -s 20251019_haveibeenpwned_sha1 hibp_sha1_dir`
|
||||
|
||||
## Prerequisites
|
||||
### Packages
|
||||
`sudo apt update && sudo apt install conv ripgrep byobu -y`
|
||||
### haveibeenpwned-downloader
|
||||
#### dotnet
|
||||
**Dotnet Installer Links**
|
||||
* [Linux Install Main Page](https://learn.microsoft.com/en-us/dotnet/core/install/linux)
|
||||
* [Debian Package Manager](https://learn.microsoft.com/en-us/dotnet/core/install/linux-debian?tabs=dotnet9)
|
||||
* [Linux Install Script](https://learn.microsoft.com/en-us/dotnet/core/install/linux-scripted-manual)
|
||||
todo: install script
|
||||
packages
|
||||
dotnet
|
||||
debian-like
|
||||
rhel-like
|
||||
fall back: manual install script
|
||||
haveibeenpwned-downloader
|
||||
maybe: dat stupid fuckoff cancer fuckin normalizefag
|
||||
todo: install instr
|
||||
todo: bashrc fuckery
|
||||
todo: archive
|
||||
todo: script to delete earlier files and softlink downdoot da latest with auto date, create new softlink
|
||||
|
||||
## Downdootan from hibp
|
||||
1. [Curl Method](https://github.com/HaveIBeenPwned/PwnedPasswordsDownloader/issues/79)
|
||||
2. [PwnedPasswordsDownloader aka haveibeenpwned-downloader](https://github.com/HaveIBeenPwned/PwnedPasswordsDownloader)
|
||||
### Commands
|
||||
make da dirrr `mkdir "$(date +%Y%m%d)_haveibeenpwned_sha1"`
|
||||
downdoot single file: `haveibeenpwned-downloader "$(date +%Y%m%d)_haveibeenpwned_sha1".tmp`
|
||||
manual nuke files `rm -f $PWD/*_hibp_sha1.tmp.txt; rm -rf $PWD/*_haveibeenpwned_sha1`
|
||||
|
||||
### hibdpdl usage
|
||||
todo: sanity checc thjessss
|
||||
one file (./myfile.txt) sha1 `haveibeenpwned-downloader myfile`
|
||||
one file (./myfile.txt) ntlm `haveibeenpwned-downloader -n myfile`
|
||||
multiple files in dir (./mydir) sha1 `haveibeenpwned-downloader mydir -s false`
|
||||
multiple filse in dir (./mydir) ntlm `haveibeenpwned-downloader -n mydir -s false`
|
||||
|
||||
## Spitballing / Scratch
|
||||
1. downdoot one larg fiel
|
||||
2. loop through five hex chars uppercase exclusive
|
||||
3. rg for those
|
||||
4. output eachi to da proper fiel :3
|
||||
|
||||
ls -q 20251019_haveibeenpwned_sha1 | head
|
||||
head 20251019_haveibeenpwned_sha1/00000.txt
|
||||
|
||||
## normalizztion
|
||||
dis silly sort FOOKAN WORTKS??
|
||||
`awk '{val="0x" $1; print strtonum(val),$0 ;}' $tmp_file | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file"`
|
||||
|
||||
timing vers wit webhookie
|
||||
`start=$(date +%s); awk '{val="0x" $1; print strtonum(val),$0 ;}' $tmp_file | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file"; end=$(date +%s); duration=$(($end - $start)); echo "$duration seconds" | tee -a time_log.txt; webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true`
|
||||
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
# Roadmap
|
||||
## Sources
|
||||
todo: links and torrents
|
||||
todo: log all acquired leaks/files
|
||||
|
||||
## Decompression and Localiztion
|
||||
decompressitalllol.sh
|
||||
log all da info
|
||||
recursively decompress to a new dir
|
||||
decompress da files in dat dir (for recursive compressions)
|
||||
log all da new file info
|
||||
find all da good files and move them into a final output dir
|
||||
|
||||
## Normalization
|
||||
todo:
|
||||
do a best to make tooling to make this a bit less manual
|
||||
convert encoding to whatever haveibeenpwned expects for sha1
|
||||
remove duplicates
|
||||
output
|
||||
sql?
|
||||
|
||||
## Checking
|
||||
test them against haveibeenpwned sha1 list
|
||||
sanity checks
|
||||
sha1 encoden each entry
|
||||
check against list
|
||||
output matches with the sha1 to good output file
|
||||
log non-matching files to non-matching output file
|
||||
ntlm list too? idk if its different data
|
||||
|
||||
## Cracking Haveibeenpwned remainders
|
||||
aws gpu node
|
||||
hashcat
|
||||
todo: define mixed attacks
|
||||
@@ -0,0 +1,208 @@
|
||||
#!/bin/bash
|
||||
|
||||
# safer bash behaviour
|
||||
set -euo pipefail
|
||||
IFS=$'\n\t'
|
||||
|
||||
# todo
|
||||
## test da shit
|
||||
## x output to output dir
|
||||
## non-interactive modes on decompress
|
||||
## handle name conflicts
|
||||
## handle password protected archives
|
||||
## handle nested archives
|
||||
## handle more archive types
|
||||
## log more info
|
||||
### tee current status to stdout and a log file with timestamps
|
||||
## get file encoding, size, type of output and log
|
||||
## generalize so i can use it in any dir
|
||||
## cmdline options for various modes
|
||||
## archive testing where possible
|
||||
## final output to another dir
|
||||
### function to normalize file encoding to whatever haveibeenpwned expects (utgf-8?)
|
||||
### recursive flattem da good files to a single dir
|
||||
## frick i ahve to delete the originaal files after backing up and decompressing them
|
||||
## because it'll break da recursive shit igg
|
||||
## fuuuuck gonna have to sha1sum every fuckin file with the file list at all stages of decompression
|
||||
### must document da info to prevent redundant effortt :pensiveanimated:
|
||||
### actually check what will be includede with torrents see if can use dat hash algo?
|
||||
## add package list
|
||||
|
||||
backup=./backup_file # backups of the archives
|
||||
log_dir=./logs # logs
|
||||
out_dir=./normalized # where the decomressed and normalizerd files go
|
||||
iterations=3 # recursive decompress levels (can override via env var ITERATIONS)
|
||||
ingress=/mnt/e/Breachsilly_Ingress # can override via env var INGRESS (use WSL-style path)
|
||||
|
||||
timestamp=$(date +%Y%m%d_%H%M%S)
|
||||
error_log="$log_dir/decompressitalllol_error_$timestamp.log"
|
||||
file_extensions="$log_dir/file_extensions_$timestamp.txt"
|
||||
file_tree="$log_dir/file_tree_$timestamp.txt"
|
||||
file_types="$log_dir/file_types_$timestamp.txt"
|
||||
|
||||
# setup
|
||||
setup_silly () {
|
||||
## maek da dirsa
|
||||
mkdir -p "$backup/tar.gz"
|
||||
mkdir -p "$backup/tar.bz2"
|
||||
mkdir -p "$backup/tar.xz"
|
||||
mkdir -p "$backup/gz"
|
||||
mkdir -p "$backup/bz2"
|
||||
mkdir -p "$backup/tar"
|
||||
mkdir -p "$backup/rar"
|
||||
mkdir -p "$backup/zip"
|
||||
mkdir -p "$backup/7z"
|
||||
mkdir -p "$backup/xz"
|
||||
mkdir -p "$log_dir"
|
||||
mkdir -p "$out_dir"
|
||||
mkdir -p "$ingress"
|
||||
}
|
||||
|
||||
# setup silly/get info
|
||||
info_silly () {
|
||||
## get a list of da file extensions
|
||||
rm -f ext.tmp || true
|
||||
while IFS= read -r -d '' f; do
|
||||
# use basename to avoid full path issues
|
||||
b="$(basename "$f")"
|
||||
# only print if there's a dot
|
||||
if [[ "$b" == *.* ]]; then
|
||||
echo "${b##*.}" >> ext.tmp
|
||||
fi
|
||||
done < <(find "$ingress" -type f -print0 2>>"$error_log")
|
||||
sort -u ext.tmp > "$file_extensions" 2>>"$error_log" || true
|
||||
rm -f ext.tmp 2>>"$error_log" || true
|
||||
|
||||
## get a list of da filetypes and encodings
|
||||
: > "$file_types"
|
||||
while IFS= read -r -d '' f; do
|
||||
file --mime-type --mime-encoding "$f" >> "$file_types" 2>>"$error_log" || true
|
||||
done < <(find "$ingress" -type f -print0 2>>"$error_log")
|
||||
|
||||
## get a simple file tree
|
||||
tree "$ingress" > "$file_tree" 2>>"$error_log" || true
|
||||
}
|
||||
|
||||
# backup all da extant archive doots
|
||||
backup_silly () {
|
||||
# get da tree and file info and suc h
|
||||
info_silly
|
||||
## tar.gz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.gz" -print -exec cp -v {} "$backup/tar.gz/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## tar.bz2
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.bz2" -print -exec cp -v {} "$backup/tar.bz2/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## tar.xz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.xz" -print -exec cp -v {} "$backup/tar.xz/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## gz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.gz" -print -exec cp -v {} "$backup/gz/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## bz2
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.bz2" -print -exec cp -v {} "$backup/bz2/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## tar
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar" -print -exec cp -v {} "$backup/tar/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## rar
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.rar" -print -exec cp -v {} "$backup/rar/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## zip
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.zip" -print -exec cp -v {} "$backup/zip/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## 7z
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.7z" -print -exec cp -v {} "$backup/7z/" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## xz
|
||||
find $ingress \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -iname "*.xz" -print -exec cp {} "$backup/xz" 2>>"$error_log" \; 2>>"$error_log"
|
||||
}
|
||||
|
||||
# recursively decompress all da things to pwd
|
||||
decompress_silly () {
|
||||
## do all the tar.x ones first to prevent issues
|
||||
### tar.gz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.gz" -print -exec tar -xvzf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### tar.bz2
|
||||
find $ingress \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -iname "*.tar.bz2" -print -exec tar -xvjf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### tar.xz
|
||||
find $ingress \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -iname "*.tar.xz" -print -exec tar -xvJf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
## now do the rest
|
||||
### gz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.gz" -print -exec bash -c 'f="$1"; out="$2"; gzip -dc "$f" > "${out}/$(basename "${f%.gz}")"' _ {} "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### bz2
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.bz2" -print -exec bash -c 'f="$1"; out="$2"; bunzip2 -c "$f" > "${out}/$(basename "${f%.bz2}")"' _ {} "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### tar
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar" -print -exec tar -xvf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### rar
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.rar" -print -exec unrar x -o+ "{}" "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### zip
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.zip" -print -exec unzip -o -d "$out_dir" "{}" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### 7z
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.7z" -print -exec 7z x "{}" -o"$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
### xz
|
||||
### test dis
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.xz" -print -exec bash -c 'f="$1"; out="$2"; unxz -c "$f" > "${out}/$(basename "$f")"' _ {} "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||
}
|
||||
|
||||
# mvoe extracted useful loookingh files to $out_dir
|
||||
move_silly () {
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -type f \( -name "*.txt" -o -name "*sql" -o -name "*csv" \) -print0 | while IFS= read -r -d '' f; do
|
||||
mv -v "$f" "$out_dir/" 2>>"$error_log" || true
|
||||
done
|
||||
}
|
||||
|
||||
# delete da originaal archives
|
||||
delete_silly () {
|
||||
## tar.gz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.gz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## tar.bz2
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.bz2" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## tar.xz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.xz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## gz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.gz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## bz2
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.bz2" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## tar
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## rar
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.rar" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## zip
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.zip" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## 7z
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.7z" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
## xz
|
||||
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.xz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||
}
|
||||
|
||||
compress_silly () {
|
||||
# compress the normalized files with 7z
|
||||
# except RockYou2024.txt and Princess_Pi*txt which are compressed with gzip
|
||||
# deletes the uncompressed output files
|
||||
## these are done with gzip because that's the compression that hashcat supports
|
||||
# gzip specific known files if they exist
|
||||
for f in "PrincessPiV8.txt" "PrincessPi_Rules_Dictionary_2.5.txt" "RockYou2024.txt"; do
|
||||
if [[ -f "$out_dir/$f" ]]; then
|
||||
gzip -9 -v "$out_dir/$f" 2>>"$error_log" || true
|
||||
fi
|
||||
done
|
||||
|
||||
# Pack remaining files (excluding already-gzipped) into a single 7z archive
|
||||
target_archive="$out_dir/normalized.7z"
|
||||
rm -f "$target_archive" || true
|
||||
find "$out_dir" -type f ! -name '*.gz' -print0 | xargs -0 -I{} 7z a -t7z "$target_archive" "{}" 2>>"$error_log" || true
|
||||
# clean up uncompressed fiies
|
||||
# delete_silly
|
||||
}
|
||||
|
||||
# run da functinoszzz
|
||||
setup_silly
|
||||
|
||||
# backup_silly
|
||||
|
||||
# decompression option
|
||||
for((i=0; i<$iterations; i++)) {
|
||||
info_silly
|
||||
decompress_silly
|
||||
}
|
||||
|
||||
# move wanted files to output dir
|
||||
move_silly
|
||||
|
||||
# decompression option
|
||||
# delete_silly
|
||||
|
||||
# compression option
|
||||
# compress_silly
|
||||
@@ -0,0 +1,145 @@
|
||||
#!/bin/bash
|
||||
set -e # dont waste my asstime if errros bitch
|
||||
# da timestiampppy liek 20251020
|
||||
timestamp=$(date +%Y%m%d)
|
||||
# fiels
|
||||
## this is both the downloaded fiel and da file to sort
|
||||
## da fuckin genetal wards dotnet app auto appends .txt to the file name jesus fuck
|
||||
download_file="lists/${timestamp}_sorted_download_single_sha1" # again no txt because the dotnet app is a form of sexually transmitted infectoion
|
||||
# count total lines for progress tracking
|
||||
echo "countin total lines..." | tee -a "$info_log"
|
||||
total_lines=$(wc -l < "$download_file.txt")
|
||||
echo "total lines to process: $total_lines" | tee -a "$info_log"
|
||||
|
||||
# for progress updates every 1%
|
||||
update_interval=$((total_lines / 100))
|
||||
[[ $update_interval -lt 1 ]] && update_interval=1
|
||||
last_update=0
|
||||
processed_lines=0
|
||||
softlink_name=hibp_list
|
||||
# backup_file=bak/test_sorted_single_sha1.tmp.txt.7z # donut need dis oc unless bein stoopd wit da backup ig
|
||||
# lougs
|
||||
info_log=info_log.log
|
||||
error_log=error_log.log
|
||||
|
||||
# ig measure how long each fluttercum takes for fun idk
|
||||
time_it () {
|
||||
echo "startin timer"
|
||||
start=$(date +%s)
|
||||
echo "runnan $1 at $(date)"
|
||||
eval "$1" 2>>"$error_log" # also make sure to get dem errorz if dey cum
|
||||
echo "done runnan $1 at $(date)"
|
||||
end=$(date +%s)
|
||||
duration=$(($end - $start))
|
||||
echo "$(date): finished $1 in $duration seconds" | tee -a "$info_log"
|
||||
}
|
||||
|
||||
break_apart () {
|
||||
# break the sortd file in ./lists into individual txt files named for the first five characters of the hashes
|
||||
echo "breakin apart da sorted fiel into smol pieces" | tee -a "$info_log"
|
||||
|
||||
# make sure hibp_sha1_dir exists and is empty
|
||||
mkdir -p hibp_sha1_dir
|
||||
rm -f hibp_sha1_dir/*.txt 2>>"$error_log"
|
||||
((processed_lines++))
|
||||
|
||||
# show progress every 1% or 100k lines
|
||||
if ((processed_lines - last_update >= update_interval)); then
|
||||
percent=$((processed_lines * 100 / total_lines))
|
||||
echo "processed $processed_lines / $total_lines lines ($percent%)" | tee -a "$info_log"
|
||||
last_update=$processed_lines
|
||||
fi
|
||||
|
||||
# read da big file line by line and split by first 5 chars
|
||||
# use a buffer to reduce disk writes - only write when prefix changes or buffer gets big
|
||||
current_prefix=""
|
||||
buffer=""
|
||||
buffer_size=0
|
||||
max_buffer=10000
|
||||
|
||||
while IFS= read -r line; do
|
||||
# get first 5 chars of the hash
|
||||
echo "finished! processed $total_lines lines into $file_count files in hibp_sha1_dir" | tee -a "$info_log"
|
||||
|
||||
# if prefix changed or buffer full, write the current buffer
|
||||
if [[ "$prefix" != "$current_prefix" ]] || ((buffer_size >= max_buffer)); then
|
||||
if [[ -n "$current_prefix" && -n "$buffer" ]]; then
|
||||
echo -n "$buffer" > "hibp_sha1_dir/${current_prefix}.txt"
|
||||
buffer=""
|
||||
buffer_size=0
|
||||
fi
|
||||
current_prefix="$prefix"
|
||||
fi
|
||||
|
||||
# add to buffer
|
||||
buffer+="${line}\n"
|
||||
((buffer_size++))
|
||||
|
||||
done < "$download_file.txt"
|
||||
|
||||
# write final buffer if any
|
||||
if [[ -n "$current_prefix" && -n "$buffer" ]]; then
|
||||
echo -n "$buffer" > "hibp_sha1_dir/${current_prefix}.txt"
|
||||
fi
|
||||
|
||||
# count how many files we made
|
||||
file_count=$(find hibp_sha1_dir -type f -name "*.txt" | wc -l)
|
||||
echo "made $file_count files in hibp_sha1_dir" | tee -a "$info_log"
|
||||
}
|
||||
|
||||
compress_run_silly () {
|
||||
# compress the files in ./lists to one 7z archive
|
||||
7z a ./hibp_sha1_individual_sorted_$timestamp.7z hibp_sha1_dir
|
||||
}
|
||||
|
||||
echo "startan scripty at $(date)"
|
||||
|
||||
# check for remaining bitchtxt and kill them if found >:34
|
||||
if [ -f lists/*.txt ]; then
|
||||
echo "i fonud sum txts in the lists dir, im nuken em lmao, list beloow" | tee -a $info_log
|
||||
ls -la lists/*.txt | tee -a $info_log
|
||||
rm -f lists/*.txt 2>> $error_log
|
||||
fi
|
||||
|
||||
# do da same for dat silly soft linkl :pope:
|
||||
if [ -f $softlink_name ]; then
|
||||
echo "fonud softline $softlink_name nukin it" | tee -a $info_log
|
||||
rm -f $softlink_name 2>> $error_log
|
||||
fi
|
||||
|
||||
# download da silly thing
|
||||
echo "startan downdoot"
|
||||
time_it "haveibeenpwned-downloader $download_file"
|
||||
|
||||
# sort it good jus to maek sure
|
||||
## dis silly widdle scrript sorts da file in place :3
|
||||
## it ver silly :3
|
||||
echo "startin da sortfuck"
|
||||
time_it "awk '{val=\"0x\" \$1; print strtonum(val),\$0 ;}' $download_file.txt | sort -n -k 1 | sed 's/^[^ ]* //'"
|
||||
|
||||
# break da file into smol pieces by hash prefix
|
||||
time_it "break_apart"
|
||||
|
||||
# zip em up heuhuehue
|
||||
compress_run_silly
|
||||
|
||||
# nao we maek dat convenient softlinkie
|
||||
# s softlink, f force overwrite existing softlink, n no dereference treat as dir
|
||||
# ls -sfn $download_file $softlink_name 2>> $error_log # for dir
|
||||
ls -sf $download_file $softlink_name 2>> $error_log # for file
|
||||
|
||||
# donut do dis when little space it get mad as fuck
|
||||
# 7z backup
|
||||
# echo "bakkan it up ig"
|
||||
# time_it "7z a $backup_file $download_file.txt"
|
||||
|
||||
# send silly webhookfuckin
|
||||
echo "sendin webhookfuckin"
|
||||
webhook "DILDO MACCARONI ITS FOOKAN DONE at $(date)" true
|
||||
|
||||
# mayhaps reboot
|
||||
## 1 minute delay to make sure shit fucks
|
||||
echo "doin an rebootasn in an 1 mins or sumtin"
|
||||
sudo shutdown -r +1 "REBOOTFUCKIN IN 10 MINS BITCH"
|
||||
|
||||
echo ":awooo: done :3"
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/bin/bash
|
||||
# todo:
|
||||
## auto byobu?
|
||||
|
||||
# to use dis i need to do checks for the dfiles that cleanup_environ cleans
|
||||
# set -e # stahp on any errpr
|
||||
|
||||
# all thos silly hex sillyvalues from 0x00000 to 0xFFFFF
|
||||
# to be used to split the hibp sha1 file into multiple files
|
||||
# each file containing all sha1 hashes starting with that sillyvalue
|
||||
dec_max=1048575
|
||||
start_val_hex=0x00000
|
||||
end_val_hex=0xFFFFF
|
||||
increment_hex=0x00001
|
||||
|
||||
# da file shit
|
||||
timestamp=$(date +%Y%m%d)
|
||||
# tmp file values
|
||||
## this is the value fed to haveibeenpwned-downloader
|
||||
## it automatically appends .txt to da filename
|
||||
tmp_file="$PWD/${timestamp}_hibp_sha1.tmp"
|
||||
## this is the txt file created by haveibeenpwned-downloader
|
||||
tmp_file_txt="$tmp_file.txt"
|
||||
new_dir="$PWD/${timestamp}_haveibeenpwned_sha1"
|
||||
softlink_name="$PWD/hibp_sha1_dir"
|
||||
softlink_target="$new_dir"
|
||||
|
||||
setup_environ () {
|
||||
# cleanup environment
|
||||
# fail quietly if no such files/dirs exist
|
||||
## nuek any temp files
|
||||
# rm -f $PWD/*_hibp_sha1.tmp.txt 2>/dev/null
|
||||
## delet any old hibp sha1 dirs
|
||||
# rm -rf $PWD/*_haveibeenpwned_sha1 2>/dev/null
|
||||
|
||||
# make new dir for new hibp sha1 files
|
||||
mkdir -p "$new_dir" # exit quietly if exists
|
||||
}
|
||||
|
||||
finish_cleanup () {
|
||||
# nuke da temp file
|
||||
rm -f "$tmp_file_txt"
|
||||
# reboot in 2 mins to clear any cached
|
||||
sudo shutdown -r +2
|
||||
}
|
||||
|
||||
downdoot_file () {
|
||||
# downdoot da fuckin file
|
||||
haveibeenpwned-downloader "$tmp_file"
|
||||
}
|
||||
|
||||
# fuckin do the hell
|
||||
do_the_fucking_split () {
|
||||
# loop throught a bajuilli9opn fuckin numbs for hex bitch
|
||||
for((i=0; i<=$dec_max; i++)) {
|
||||
current_hex=$(printf "0x%05X" $i)
|
||||
echo "$current_hex"
|
||||
|
||||
# do da fuckin hunt and shit fuckbitchfuck
|
||||
rg --no-line-number "^$current_hex" "$tmp_file_txt" > "$new_dir/$current_hex.txt"
|
||||
}
|
||||
}
|
||||
|
||||
post_process_files () {
|
||||
# softlink
|
||||
# s softlink, f force overwrite existing softlink, n no dereference treat as dir
|
||||
ln -sfn "$softlink_target" "$softlink_name"
|
||||
}
|
||||
|
||||
setup_environ
|
||||
downdoot_file
|
||||
do_the_fucking_split
|
||||
post_process_files
|
||||
# finish_cleanup
|
||||
@@ -0,0 +1,2 @@
|
||||
#!/bin/bash
|
||||
smbencrypt dingle | awk '{print $1}' | tail -1
|
||||
@@ -0,0 +1,2 @@
|
||||
#!/bin/bash
|
||||
echo "$1" | sha1sum | awk '{print $1}' | tr 'a-z' 'A-Z'
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/bin/bash
|
||||
# fiels
|
||||
input_file=./test_sha1.tmp.txt
|
||||
sorted_file=./test_SORTED_sha1.tmp.txt
|
||||
backup_file=./bak/20251020_single_sha1.txt.7z
|
||||
# lougs
|
||||
info_log=./info_log.log
|
||||
error_log=./error_log.log
|
||||
|
||||
# start da timer
|
||||
echo "startin timer"
|
||||
start=$(date +%s)
|
||||
|
||||
# do da magical sortfuck
|
||||
echo "performin da magical sortfuck"
|
||||
# awk '{val="0x" $1; print strtonum(val),$0 ;}' "$input_file" | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file" 2>> "$error_log"
|
||||
awk '{val="0x" $1; print strtonum(val),$0 ;}' $input_file | sort -n -k 1 | sed 's/^[^ ]* //' > $sorted_file
|
||||
echo "finished da magical sortfuck! retcode $?"
|
||||
|
||||
# end da timer and caluclate duratrion
|
||||
echo "endin timer"
|
||||
end=$(date +%s)
|
||||
duration=$(($end - $start))
|
||||
echo "sorted (in theoty) in $duration seconds"
|
||||
|
||||
# updoot da log
|
||||
## can reuse dis in a function or sumshit
|
||||
echo "loggin shit"
|
||||
echo "$(date): $input_file sorted to $sorted_file in $duration seconds" | tee -a "$info_log"
|
||||
|
||||
# send silly webhookfuckin
|
||||
echo "sendin webhookfuckin"
|
||||
webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true
|
||||
|
||||
# maek an backup
|
||||
echo "maekan a backup"
|
||||
7z a "$backup_file" "$input_file"
|
||||
|
||||
# mayhaps reboot
|
||||
## 10 minute delay to make sure shit fucks
|
||||
echo "doin an rebootasn in 10 mins or sumtin"
|
||||
sudo shutdown -r +10 "REBOOTFUCKIN IN 10 MINS BITCH"
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
# fiels
|
||||
input_file=./test_sha1.tmp.txt
|
||||
sorted_file=./test_SORTED_sha1.tmp.txt
|
||||
# lougs
|
||||
info_log=./info_log.log
|
||||
error_log=./error_log.log
|
||||
|
||||
# maek an backup
|
||||
echo "maekan a backup"
|
||||
cp "$input_file" "$input_file.bak"
|
||||
|
||||
# start da timer
|
||||
echo "startin timer"
|
||||
start=$(date +%s)
|
||||
|
||||
# do da magical sortfuck
|
||||
echo "performin da magical sortfuck"
|
||||
awk '{val="0x" $1; print strtonum(val),$0 ;}' "$input_file" | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file" 2>> "$error_log"
|
||||
echo "finished da magical sortfuck! retcode $?"
|
||||
|
||||
# end da timer and caluclate duratrion
|
||||
echo "endin timer"
|
||||
end=$(date +%s)
|
||||
duration=$(($end - $start))
|
||||
echo "sorted (in theoty) in $duration seconds"
|
||||
|
||||
# updoot da log
|
||||
## can reuse dis in a function or sumshit
|
||||
echo "loggin shit"
|
||||
echo "$(date): $input_file sorted to $sorted_file in $duration seconds" | tee -a "$info_log"
|
||||
|
||||
# send silly webhookfuckin
|
||||
echo "sendin webhookfuckin"
|
||||
webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true
|
||||
|
||||
# mayhaps reboot
|
||||
## 10 minute delay to make sure shit fucks
|
||||
echo "doin an rebootasn in 10 mins or sumtin"
|
||||
sudo shutdown -r +10 "REBOOTFUCKIN IN 10 MINS BITCH"
|
||||
@@ -0,0 +1,3 @@
|
||||
tar gzip bzip2 rar zip 7z xz conv ripgrep screen byobu
|
||||
dotnet
|
||||
haveibeenpwned-downloader
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
IFS=$'\n\t'
|
||||
|
||||
# Default settings
|
||||
output_encoding="utf-8"
|
||||
hibp_sha1_dir="${HIBP_SHA1_DIR:-hibp_sha1_dir}" # Can override with env var
|
||||
|
||||
# Check for required tools
|
||||
for cmd in file iconv sha1sum rg; do
|
||||
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||
echo "Error: Required command '$cmd' not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
# Validate arguments
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo "Usage: $(basename "$0") <password> [debug]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
input_str="$1"
|
||||
debug_mode=0
|
||||
[[ ${2:-} == "debug" ]] && debug_mode=1
|
||||
|
||||
# Validate HIBP directory exists
|
||||
if [[ ! -d "$hibp_sha1_dir" ]]; then
|
||||
echo "Error: HIBP SHA1 directory '$hibp_sha1_dir' not found" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# detect da input encoding
|
||||
input_encoding="$(echo -n "$input_str" | file -i - | awk -F'charset=' '{print $2}' || echo 'utf-8')"
|
||||
[[ -z "$input_encoding" ]] && input_encoding="utf-8"
|
||||
|
||||
# convert da silly into utf-8
|
||||
converted_str=$(iconv -f "$input_encoding" -t "$output_encoding" <<< "$input_str" 2>/dev/null || echo "$input_str")
|
||||
|
||||
# convert da silly into sha1
|
||||
sha1_str=$(echo "$converted_str" | sha1sum | awk '{print $1}' | tr 'a-z' 'A-Z')
|
||||
|
||||
# get da first five chars
|
||||
first_five="${sha1_str:0:5}"
|
||||
|
||||
# lookup da hash in hibp sha1 dir
|
||||
lookup_file="$hibp_sha1_dir/$first_five.txt"
|
||||
|
||||
if ((debug_mode)); then
|
||||
echo "input_str (\$1): $input_str"
|
||||
echo "hibp_sha1_dir: $hibp_sha1_dir"
|
||||
echo "output_encoding: $output_encoding"
|
||||
echo "input_encoding: $input_encoding"
|
||||
echo "converted_str: $converted_str"
|
||||
echo "sha1_str: $sha1_str"
|
||||
echo "first_five: $first_five"
|
||||
echo "lookup_file: $lookup_file"
|
||||
fi
|
||||
|
||||
# Check if lookup file exists
|
||||
if [[ ! -f "$lookup_file" ]]; then
|
||||
echo "Hash prefix $first_five not found in database" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Look up the hash (ripgrep returns 0 if found, 1 if not found)
|
||||
if rg --quiet --fixed-strings "$sha1_str" "$lookup_file" 2>/dev/null; then
|
||||
echo "Password found in breaches"
|
||||
exit 0
|
||||
else
|
||||
echo "Password not found in breaches"
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,25 @@
|
||||
awk '{val="0x" $1; print strtonum(val),$0 ;}' your_file.txt | sort -n -k 1 | sed 's/^[^ ]* //'
|
||||
|
||||
```
|
||||
#!/bin/bash
|
||||
# fiels
|
||||
input_file="$PWD/test_sha1.tmp.txt"
|
||||
sorted_file="$PWD/test_SORTED_sha1.tmp.txt"
|
||||
log_file="$PWD/info.log"
|
||||
|
||||
# start da timer
|
||||
start=$(date +%s)
|
||||
|
||||
# do da magical sortfuck
|
||||
awk '{val="0x" $1; print strtonum(val),$0 ;}' "$input_file" | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file"
|
||||
end=$(date +%s)
|
||||
|
||||
# end da timer and caluclate duratrion
|
||||
duration=$(($end - $start))
|
||||
|
||||
# updoot da log
|
||||
echo "$(date): $input_file sorted to $sorted_file in $duration seconds" | tee -a "$log_file"
|
||||
|
||||
# send silly webhookfuckin
|
||||
webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true
|
||||
```
|
||||
@@ -0,0 +1,30 @@
|
||||
auto start in byobu
|
||||
environment check
|
||||
packages
|
||||
dotnet
|
||||
downdooter
|
||||
files in current dir
|
||||
interactively ask to delete files
|
||||
speed up rg
|
||||
test time
|
||||
nosleep
|
||||
sanity checks
|
||||
hash lengths
|
||||
successful sorting in txt files
|
||||
sum kinda bettar progress indicator
|
||||
timing for estimated completion
|
||||
|
||||
sort then split! # literally fucking must do this to have any god damned chance
|
||||
# DO SOME MORE RESEARCH
|
||||
# also prepends da 0x remov
|
||||
awk '{val="0x" $1; print strtonum(val),$0 ;}' your_file | sort -n -k 1 | sed 's/^[^ ]* //'
|
||||
|
||||
# another option
|
||||
## does this assume 0x??
|
||||
paste -d: <(echo 'ibase=16'; cut -d' ' -f1 your_file | bc) your_file | sort -t: -k1,1n | cut -d: -f2-
|
||||
|
||||
# opttion?
|
||||
## prepend 0x to each line?
|
||||
## -g is GENRAL numerical sort and it can handle hex nativelu
|
||||
## does not work will have to prepend 0x
|
||||
sort -o outfilename -g filename
|
||||
Reference in New Issue
Block a user