Investigative Data Warehouse

The Investigative Data Warehouse (IDW) is a searchable database operated by the FBI . It was created in 2004. Much of the nature and scope of the database is classified . The database is a centralization of multiple federal and state databases, Including criminal records from various law enforcement agencies , the US Department of the Treasury ‘s Financial Crimes Enforcement Network (FinCEN), and public records databases. According to Michael Morehart’s testimony before the House Committee on Financial Services in 2006, the “IDW is a centralized, web-enabled, Closed system repository for intelligence and investigative data. This system, maintained by the FBI, permits the appropriately trained and authorized personnel throughout the country to query for information of relevance to investigative and intelligence matters. ” [1]


The size of the database will appear rapidly. In 2004, according to government solicitation for bids to manage the project, it was approximately 10 TB in size. In 2005, according to one FBI official, the IDW contained approximately 100 million documents. In 2006 it contained more than 560 million documents and was accessible by more than 12,000 individuals. According to the FBI website, as of August 22, 2007, the database contained 700 million records from 53 databases and was accessed by 13,000 individuals around the world.

As of 2007 , the FBI is the subject of a lawsuit Brought by the EFF ( Electronic Frontier Foundation ) Because of a Lack of public instruction Describing the database and the criteria for Including personal information, as required by the Privacy Act of 1974 . The lawsuits are a result of two Freedom of Information Act requests filed by the EFF in 2006.

It was built in part by Chiliad Corporation, [2] [3] the FBI Office of the Chief Technology Officer, [4] and others. Companies listed on the FOIA files include Northrup Grumman [5] and others.


Investigative Data Warehouse-Secret, “provides data and data processing services to FBI agents and analysts as they perform counter-terrorism, counter-intelligence, and enforcement missions.” The Core Subsystem supports the Counter-Terrorism Division (CTD), the Special Event Unit, and DOCLAB-S, the Joint Intelligence Committee Investigation (JICI) and IntelPlus. [6]

According to a 2005 email, “IDW will also be used for criminal and other non-CT investigations as it evolves.” (CT being Counter Terrorism) [7]


Within the system, there were subsystems named IDW-S Core, SPT, and DOCLAB-S [8]


SPT stands for Special Projects Team. It

(S) of the. These data sources are not necessarily available to the general IDW users but only to a small group of users who have a demonstrated “need-to-know”. The SPT System is similar in function to the IDW-S system, with the main difference being a different set of data sources. The SPT System allows users to access the IDT Data Store. [9]


According to FBI, the Privacy Impact Assessments (PIAs) of the IDW system. They Worked with lawyers from Their National Security Law Branch (NSLB) to attempt to make sour Their system Was Complying with various laws Regarding sharing of information and secrecy [10] (for example, rule 6 of the Federal Rules of Criminal Procedure, Regarding the Secrecy of Grand Jury material [11] ).

The Information Sharing Policy Group (ISPG) is a discretionary Access Control Team (DACT) for IDW and DataMart, and responds to other Intelligence Community agencies requesting access. [12]

The EFF FOIA IDW website states “Despite the vast amount of personal information contained in the IDW, the FBI has never published a privacy statement describing the system or explaining the ways in which the records might be used.” [13]

There was also an email from someone on the Office of General Council (OGC) about “FBI PIA requirements to non-NS systems” (NS being National Security). [14] There Was aussi an email from 2006 Saying that ‘national security systems are free from E-Gov “, [15] Apparently referring to the E-Government Act of 2002 , section: has a qui That deals with privacy.

Data sources

The IDW used many data sources. The FOIA documents from EFF are heavily redacted, but some of the sources are as follows:

  • FBI Automated Case Support System (ACS), subset of the Electronic Case File (ECF) system [16]
  • Joint Intelligence Committee Investigation Documents (JICI), [17] with OCR text [18]
  • “Open Source News” (public websites, Such As the Washington Post and others) [19]
  • Secure Automated Messaging Network (SAMNet) [17]
  • Violent Gang and Terrorist Organizing File (VGTOF)
  • DARPA TIDES program (‘open source news’ that has been organized and collected)
  • IntelPlus Filerooms, with OCR text [18]
  • FBI National Crime Information Center (NCIC) [16]
  • FBI Records Management Division (RMD), Document Laboratory (DocLab), FBIHQ
  • MiTAP [20] (collects data from public sources, websites, etc.)
  • SPT-Specific data sources (partial list, FOIA files have large parts redacted):
    • Unified Name Index (UNI) extracts
    • Financial Center (FinCen), [21] including Bank Secrecy Act data [22]
    • “Various Sources”, including the Transportation Security Administration [23]
    • FBI Counterterrorism Division (CTD)
    • Telephone numbers / addresses from ACS
    • Case data from ACS
    • Terrorist Watch List (TWL) [24]
    • “Other NJTTF data”
    • DoS … Lost / Stolen Passport data
    • No Fly List , from TSA
    • Selectee list, from TSA
    • ACS / ECF with some cases types excluded
    • CIA Non-TS / Non-SCI Technical Discussions (TDs) and Intelligence Information Reports (IIRs) from 1978 to the May 2004 [25]

There was also talk of linking the FTTTF “Data Mart” with IDW. [26]

The data in IDW is classified at the ‘Secret’ level or lower. Higher classifications are not allowed, and can be removed [27]

See also

  • Open-source intelligence
  • FBI Index
  • DCSNet
  • Data Loading and Analysis System
  • Data Warehouse System Electronic Surveillance Data Management System
  • Project SHAMROCK (NSA)
  • Project MINARET (NSA)


Sources consulted
  • EFF. “FOIA: DOJ’s Investigative Data Warehouse” . Retrieved 2009-03-18 .
  • EFF (October 17, 2006). “EFF Sues for Information on Huge FBI Database of Personal Information: ‘Investigative Data Warehouse’ Includes Hundreds of Millions of Entries” . Retrieved 2009-03-17 .
  • FBI (various dates). “EFF Freedom of Information Act (FOIA) files, 2008 April 8, idw02” (PDF) . Electronic Frontier Foundation . Retrieved 2009-03-17 . Check out dates in: ( help ) (Contains various emails from inside the FBI regarding the IDW) |date=
  • FBI (various dates). “EFF Freedom of Information Act (FOIA) Files, 2008 Apr 8, idw01” (PDF) . Electronic Frontier Foundation . Retrieved 2009-03-18 . Check out dates in: ( help ) (Contains various emails from inside the FBI regarding the IDW) |date=
  • FBI (various dates). “EFF Freedom of Information Act (FOIA) files, 2008 June 9, idw04” (PDF) . Electronic Frontier Foundation . Retrieved 2009-03-18 . Check out dates in: ( help ) (Contains various emails from inside the FBI regarding the IDW) |date=
  • FBI (Sep 6, 2006). “By the Numbers: FBI Transformation Since 2001” . Retrieved 2009-03-17 .
  • FBI Information Resources Division (IRD) (2003-12-03). “Investigative Data Warehouse-SECRET (IDW-S), System Security Plan” (PDF) . Electronic Frontier Foundation. p. 58 . Retrieved 2009-03-17 .
  • FBI Information Resources Division, Data and Information Management Section (2005-01-24). “Investigative Data Warehouse – Secret (IDW-S), System Security Plan” (PDF) . Electronic Frontier Foundation. p. 13 . Retrieved 2009-03-18 .
  • FBI Information Resources Division, Data Management Section (2004-12-01). “Investigative Data Warehouse, Privileged Users Guide” (PDF) . Electronic Frontier Foundation. p. 9 . Retrieved 2009-03-18 .
  • FBI Office of the Program Management Executive (2004-11-29). “Security Concept of Operations (S-CONOPS), Investigative Data Warehouse (IDW) Program” (PDF) . Electronic Frontier Foundation. p. 50.
  • Gross, Grant (17 October 2006). “EFF files lawsuit to gain information on FBI database” . Network World (reprinting IDG News Service story) .
  • Morehart, Michael FA (May 26, 2005). “Statement of Michael FA Morehart, Section Chief, Terrorist Financing Operations Section, Counterterrorism Division, Federal Bureau of Investigation, Before the House Committee on Financial Services . ” Retrieved 2009-03-17 .
  • Nakashima, Ellen (30 August 2006). “FBI Shows Off Counterterrorism Database” . The Washington Post .
  1. Jump up^ Morehart 2005, op. cit.
  2. Jump up^ “Chiliad Case Study” (PDF) . Retrieved 2009-03-18 .
  3. Jump up^ David Gardner (2006-08-30). “FBI Shows off Counterterrorism Database” . Information Week . Retrieved 2009-03-18 .
  4. Jump up^ EFF FOIA Files, 2008 Apr 8, idw01, page 28 of linked pdf
  5. Jump up^ EFF FOIA files, 2008 Apr 8 idw01, page 27 of linked pdf
  6. Jump up^ FBI, IDW-S System Security Plan, 2005 Jan 24
  7. Jump up^ EFF FOIA files, 2008 Apr 8 idw02, pg 13 of linked PDF
  8. Jump up^ FBI, IDW-S System Security Plan, 2005 Jan. 24. It is unclear from the FOIA documents the difference between IDW-S and IDW, and the Core SPT and DOCLAB-S are under IDW, or IDW-S.
  9. Jump up^ FBI, S-CONOPS IDW, 2004 Nov 29page 52 of linked pdf
  10. Jump up^ EFF FOIA Files, 2008 April 8 idw02. Most of this FOIA release is emails within the FBI about PIAs
  11. Jump up^ EFF FOIA Files, 2008 April 8 idw02, page 73 of linked pdf. For Rule 6e, see
  12. Jump up^ EFF FOIA Files, 2008 April 8 idw02pg 74, 75 of linked pdf
  13. Jump up^ EFF website, FOIA: DOJ’s Investigative Data Warehouse
  14. Jump up^ EFF FOIA Files, 2008 April 8 idw02, page 10 of linked pdf. This particular e-mail also has theVCFsystem (which was later scrapped), saying that PIAs for VCF could ‘entail substantial costs’
  15. Jump up^ EFF FOIA Files, 2008 Jun 9 idw04, page 35 of linked pdf
  16. ^ Jump up to:b FBI, IDW Privileged Users Guide, 2004 Dec 1
  17. ^ Jump up to:b FBI, IDW-S System Security Plan, 2003 Dec 3
  18. ^ Jump up to:b FBI IDW Status Update, 2005 Sep 21
  19. Jump up^ FBI IDW Status Update, 2005 Sep 21. ‘Open Source News’ is, in other documents, referred to alongsideMiTAPand / or DARPA TIDES.
  20. Jump up^ Note: Some FBI documents list DARPA TIDES, some list MiTAP, some simply say “Open Source News”. They are related projects, if not even the same thing.
  21. Jump up^ Financial Crimes Enforcement Network
  22. Jump up^ EFF FOIA files 2008 Apr 8 idw02, pg 8/9 of linked pdf
  23. Jump up^ FBI S-CONOPS IDW 2004 Nov 29page 53 of linked pdf
  24. Jump up^ EFF FOIA Files, 2008 Apr 8, idw02page 83 of linked PDF
  25. Jump up^ EFF FOIA Files, 2008 Apr 8, idw01, page 33 of linked pdf
  26. Jump up^ EFF FOIA Files, 2008 Apr 8, idw02. Page 37 of linked pdf
  27. Jump up^ EFF FOIA files, 2008 Apr 2, idw01, page 43

Leave a Comment

Your email address will not be published. Required fields are marked *