AN11--Kaiser Permanente R&D Statistical Expertise EHR Evaluation

2 Notice of Sources Sought The purpose of this Sources Sought Announcement is for market research to make appropriate acquisition decisions and to gain knowledge of potential qualified Service-Disabled Veteran Owned Small Businesses (SDVOSB), Veteran Owned Small Businesses (VOSB), Women Owned Small Businesses (WOSB) and Small Businesses interested and capable of providing the services requested, as well as any large businesses. The results from this source sought notice will be used to determine the appropriate set-aside. The intended contract is a firm-fixed price R&D service contract. The SBA Non-Manufacturer Rule is not applicable as the intended contract is for services. The Department of Veterans Affairs, NCO 20 is looking for Kaiser Permanente s authorized service providers that can provide below R&D services. Statement of Work: Contract Title Expertise in Measurement Error to Support VHA Digital Health Office Immersive Program Evaluation Project: VHA Digital Health Office Immersive Program Evaluation Background Virtual reality (VR) is an emerging technology that has been used as a treatment in diverse pain populations and for diverse indications. VA's Digital Health Office is exploring offering Veterans a variety of VR therapies through conducting multiple pilot initiatives. To date over 8,000 Veterans have engaged in VR activities through engaging in headset technologies in clinics and at home. Quantifying the potential benefit of disseminating VR technologies requires careful assessment of the VR activities, how they are being used in combination with other therapies such as mental health treatment and physical therapy, and monitoring of outcomes including patient reported changes in health, pain and well-being, healthcare utilization changes and overall costs. The Digital Health Office is providing tools to measure VR engagement and has requested evaluation of the VHA Immersive Program and to generate insights about how to best support further XR implementation. The evaluation activities will leverage data collected as part of VHA Immersive Program activities, data about participants linked to VHA administrative data, and primary data collected by the evaluation team. The evaluation will span three fiscal years, and will be comprised of retrospective analyses, mixed-methods evaluation design, and prospective data collection. To understand how population-level coding and documentation of VR engagement reflect true exposure levels, fidelity, and overall VR dose we require a statistical expert. We are requesting a contract with Kaiser Permanente Washington Health Research Institute (KPWHRI) in order for Pamela Shaw, PhD to work with this project and provide expertise in statistical methods and study design. Dr. Shaw is a Senior Investigator in the Division of Biostatistics at KPWHRI and also Affiliate Professor in the Department of Biostatistics at the University of Washington School of Public Health. Dr. Shaw s research focuses on the design and analysis of studies leveraging electronic health records (EHR) data to support rigorous, cost-effective, clinical and epidemiologic research to improve the health and guide treatment decisions for US citizens. Dr. Shaw brings expertise in complex survey design, statistical methods and study design to address measurement error and outcome misclassification, and the evaluations of interventions using clinical trial and observational data across a variety of health care settings and populations. Her work integrates methodological rigor with practical relevance, advancing the use of electronic health records data to public health and clinical decision making. Selected research projects done under Dr. Shaw s leadership: Williamson B, Krakauer C, Johnson E, Gruber S, Shepherd BE, van der Laan MJ, Lumley T, Lee H, Hernandez Munoz JJ, Zhao F, Dutcher SK, Desai, R, Simon GE, Shortreed SM, Nelson JC, Shaw PA. Assessing treatment effects in observational data with missing confounders: A comparative study of practical doubly-robust and traditional missing data methods. Statistics in Medicine, 2026 Feb;45(3-5):e70366 Shepherd BE, Han K, Chen T, Bian A, Pugh S, Duda SN, Lumley T, Heerman WJ, Shaw PA. Multiwave validation sampling for error-prone electronic health records. Biometrics. 2023 Sep;79(3):2649-2663. PubMed Central PMCID: PMC10525037. Shaw PA, Yang JB, Mowery DL, Schriver ER, Mahoney KB, Bar KJ, Ellenberg SS. Determinants of hospital outcomes for COVID-19 infections in a large Pennsylvania Health System. PLoS ONE, 2022 May 19; 17(5): e0268528. Shaw PA, Gustafson P, Carroll RJ, Deffner V, Dodd KW, Keogh RH, Kipnis V, Tooze JA, Wallace MP, Kuchenhoff H, Freedman LS. STRATOS guidance document on measurement error and misclassification of variables in observational epidemiology: Part II -more complex methods of adjustment and advanced topics.  Statistics in Medicine 2020 Jul 20;39(16):2232-2263.  Han K, Lumley T, Shepherd BE, Shaw PA. Two-phase analysis and study design for survival models with error-prone exposures. Stat Methods Med Res. 2020 Dec 16; PubMed Central PMCID: PMC8715910. Shaw PA. Use of composite outcomes to assess risk-benefit in clinical trials. Clin Trials. 2018 Aug;15(4):352-358. PubMed PMID: 30021496. Scope Dr. Shaw will provide expertise in measurement error and missing data in studies of longitudinal EHR outcomes following the rigorous frameworks for designing and analyzing observations studies as outlined in the international STRATOS guidance about statistical methods to minimize measurement error and her previous work on two-phase analyses and study designs to address error-prone electronic health records data. This will include reviewing study protocols and statistical analysis plans that rely on EHR data that are subject to missingness and measurement error, especially about use of VR/Immersive technologies that are utilized through a combination of in-clinic and at-home sessions. Dr. Shaw will also provide specific expertise in designing future prospective study protocols and data collection strategies for future evaluations of VR/Immerstive technologies that address potential residual sources of bias associated with measurement error and missing data. Dr. Shaw will work with the analytics team to review available covariate information for patients engaging in VR therapies. Based on observed patterns, she will support two efforts: 1) Provide guidance for prospective data collection efforts to improve precision in specific sub-populations about factors most likely to influence the causal quantities of interest. 2) Provide guidance regarding robust and efficient analytic approaches that leverage auxiliary EHR data with outcome data only available on a subset (e.g. patent reported outcomes). This contract will strengthen analytic accuracy, efficiency, and the credibility of evaluation outcomes. Tasks and Deliverables Task 1: Collaborate with VA Puget Sound study team to develop study protocols and statistical analysis plans. Ensure development of any necessary statistical analysis plans are in line with STRATOS/STROBE guidelines. Task 2: Review EHR outcome definitions, data extraction algorithms, and subset of patient reported outcome data available from specific immersive pilot Task 3: Guide sampling procedures for future prospective data collection efforts among patient subgroups for specific immersive pilot initiatives to supplement EHR outcomes with supplementally collected patient reported outcomes. Task 4: Code review of VA Puget Sound study analysts and replication of analytic findings to ensure reproducibility and quality control. Task 5: Feedback and review of VA Puget Sound analytic team technical reports detailing internal evaluation findings. Some internal reports may be of interest to broader healthcare community and, if appropriate, will be prepared for peer-review and dissemination through journal publications. Dr. Shaw will serve as a methodologic expert and co-author in manuscript preparation. Deliverable Description Delivery Date 1 Development of study protocols and statistical analysis plans in accordance with STRATOS/STROBE guidelines. As needed 2 Review of EHR outcome definitions, data extraction algorithms, and available patient-reported outcome data from immersive pilot initiatives. As needed 3 Provide sampling recommendations for prospective patient subgroup data collection efforts to supplement EHR outcomes with patient-reported outcomes. As needed 4 Code review and independent replication of analytic findings with written documentation of review outcomes. As needed 5 Review of and feedback on internal technical reports; co-authorship of manuscripts prepared for peer-review submission, as appropriate. As needed 6 Virtual and in-person meeting attendance. As needed Performance Monitoring Dr. Zeliadt will provide oversight of the work, which will be measured by meeting project deadlines in a timely fashion and regularly assessed quality of the work in achieving the stated project goals. This regular performance evaluation will be recorded in minutes that will be available to the Station POC/COR for review. The Station POC/COR will work with Dr. Zeliadt and the evaluation team to complete the required annual performance evaluation. Place of Performance The work will be conducted remotely. Dr. Shaw will also attend in-person meetings at the VA Puget Sound Health Care System. Period of Performance Base year plus two option years. The anticipated start date is September 15, 2026. VA INFORMATION AND INFORMATION SYSTEM SECURITY/PRIVACY LANGUAGE FOR INCLUSION INTO CONTRACTS, AS APPROPRIATE GENERAL Contractors, contractor personnel, subcontractors, and subcontractor personnel shall be subject to the same Federal laws, regulations, standards, and VA Directives and Handbooks as VA and VA personnel regarding information and information system security. ACCESS TO VA INFORMATION AND VA INFORMATION SYSTEMS A contractor/subcontractor shall request logical (technical) or physical access to VA information and VA information systems for their employees, subcontractors, and affiliates only to the extent necessary to perform the services specified in the contract, agreement, or task order. All contractors, subcontractors, and third-party servicers and associates working with VA information are subject to the same investigative requirements as those of VA appointees or employees who have access to the same types of information. The level and process of background security investigations for contractors must be in accordance with VA Directive and Handbook 0710, Personnel Suitability and Security Program. The Office for Operations, Security, and Preparedness is responsible for these policies and procedures. Contract personnel who require access to national security programs must have a valid security clearance. National Industrial Security Program (NISP) was established by Executive Order 12829 to ensure that cleared U.S. defense industry contract personnel safeguard the classified information in their possession while performing work on contracts, programs, bids, or research and development efforts. The Department of Veterans Affairs does not have a Memorandum of Agreement with Defense Security Service (DSS). Verification of a Security Clearance must be processed through the Special Security Officer located in the Planning and National Security Service within the Office of Operations, Security, and Preparedness. Custom software development and outsourced operations must be located in the U.S. to the maximum extent practical. If such services are proposed to be performed abroad and are not disallowed by other VA policy or mandates, the contractor/subcontractor must state where all non-U.S. services are provided and detail a security plan, deemed to be acceptable by VA, specifically to address mitigation of the resulting problems of communication, control, data protection, and so forth. Location within the U.S. may be an evaluation factor. 3 The contractor or subcontractor must notify the Contracting Officer immediately when an employee working on a VA system or with access to VA information is reassigned or leaves the contractor or subcontractor s employ. The Contracting Officer must also be notified immediately by the contractor or subcontractor prior to an unfriendly termination. VA INFORMATION CUSTODIAL LANGUAGE Information made available to the contractor or subcontractor by VA for the performance or administration of this contract or information developed by the contractor/subcontractor in performance or administration of the contract shall be used only for those purposes and shall not be used in any other way without the prior written agreement of the VA. This clause expressly limits the contractor/subcontractor's rights to use data as described in Rights in Data - General, FAR 52.227-14(d) (1). VA information should not be co-mingled, if possible, with any other data on the contractors/subcontractor s information systems or media storage systems in order to ensure VA requirements related to data protection and media sanitization can be met. If co-mingling must be allowed to meet the requirements of the business need, the contractor must ensure that VA s information is returned to the VA or destroyed in accordance with VA s sanitization requirements. VA reserves the right to conduct on site inspections of contractor and subcontractor IT resources to ensure data security controls, separation of data and job duties, and destruction/media sanitization procedures are in compliance with VA directive requirements. Prior to termination or completion of this contract, contractor/subcontractor must not destroy information received from VA, or gathered/created by the contractor in the course of performing this contract without prior written approval by the VA. Any data destruction done on behalf of VA by a contractor/subcontractor must be done in accordance with National Archives and Records Administration (NARA) requirements as outlined in VA Directive 6300, Records and Information Management and its Handbook 6300.1 Records Management Procedures, applicable VA Records Control Schedules, and VA Handbook 6500.1, Electronic Media Sanitization. Self-certification by the contractor that the data destruction requirements above have been met must be sent to the VA Contracting Officer within 30 days of termination of the contract. The contractor/subcontractor must receive, gather, store, back up, maintain, use, disclose and dispose of VA information only in compliance with the terms of the contract and applicable Federal and VA information confidentiality and security laws, regulations and policies. If Federal or VA information confidentiality and security laws, regulations and policies become applicable to the VA information or information systems after execution of the contract, or if NIST issues or updates applicable FIPS or Special Publications (SP) after execution of this contract, the parties agree to negotiate in good faith to implement the information confidentiality and security laws, regulations and policies in this contract. The contractor/subcontractor shall not make copies of VA information except as authorized and necessary to perform the terms of the agreement or to preserve electronic information stored on contractor/subcontractor electronic storage media for restoration in case any electronic equipment or data used by the contractor/subcontractor needs to be restored to an operating state. If copies are made for restoration purposes, after the restoration is complete, the copies must be appropriately destroyed. If VA determines that the contractor has violated any of the information confidentiality, privacy, and security provisions of the contract, it shall be sufficient grounds for VA to withhold payment to the contractor or third party or terminate the contract for default or terminate for cause under Federal Acquisition Regulation (FAR) part 12. If a VHA contract is terminated for cause, the associated BAA must also be terminated and appropriate actions taken in accordance with VHA Handbook 1600.01, Business Associate Agreements. Absent an agreement to use or disclose protected health information, there is no business associate relationship. The contractor/subcontractor must store, transport, or transmit VA sensitive information in an encrypted form, using VA-approved encryption tools that are, at a minimum, FIPS 140-2 validated. The contractor/subcontractor s firewall and Web services security controls, if applicable, shall meet or exceed VA s minimum requirements. VA Configuration Guidelines are available upon request. Except for uses and disclosures of VA information authorized by this contract for performance of the contract, the contractor/subcontractor may use and disclose VA information only in two other situations: (i) in response to a qualifying order of a court of competent jurisdiction, or (ii) with VA s prior written approval. The contractor/subcontractor must refer all requests for, demands for production of, or inquiries about, VA information and information systems to the VA contracting officer for response. Notwithstanding the provision above, the contractor/subcontractor shall not release VA records protected by Title 38 U.S.C. 5705, confidentiality of medical quality assurance records and/or Title 38 U.S.C. 7332, confidentiality of certain health records pertaining to drug addiction, sickle cell anemia, alcoholism or alcohol abuse, or infection with human immunodeficiency virus. If the contractor/subcontractor is in receipt of a court order or other requests for the above mentioned information, that contractor/subcontractor shall immediately refer such court orders or other requests to the VA contracting officer for response. 5 4 For service that involves the storage, generating, transmitting, or exchanging of VA sensitive information but does not require C&A or an MOU-ISA for system interconnection, the contractor/subcontractor must complete a Contractor Security Control Assessment (CSCA) on a yearly basis and provide it to the COTR. INFORMATION SYSTEM DESIGN AND DEVELOPMENT Information systems that are designed or developed for or on behalf of VA at non-VA facilities shall comply with all VA directives developed in accordance with FISMA, HIPAA, NIST, and related VA security and privacy control requirements for Federal information systems. This includes standards for the protection of electronic PHI, outlined in 45 C.F.R. Part 164, Subpart C, information and system security categorization level designations in accordance with FIPS 199 and FIPS 200 with implementation of all baseline security controls commensurate with the FIPS 199 system security categorization (reference Appendix D of VA Handbook 6500, VA Information Security Program). During the development cycle a Privacy Impact Assessment (PIA) must be completed, provided to the COTR, and approved by the VA Privacy Service in accordance with Directive 6507, VA Privacy Impact Assessment. The contractor/subcontractor shall certify to the COTR that applications are fully functional and operate correctly as intended on systems using the VA Federal Desktop Core Configuration (FDCC), and the common security configuration guidelines provided by NIST or the VA. This includes Internet Explorer 7 configured to operate on Windows XP and Vista (in Protected Mode on Vista) and future versions, as required. The standard installation, operation, maintenance, updating, and patching of software shall not alter the configuration settings from the VA approved and FDCC configuration. Information technology staff must also use the Windows Installer Service for installation to the default program files directory and silently install and uninstall. Applications designed for normal end users shall run in the standard user context without elevated system administration privileges. The security controls must be designed, developed, approved by VA, and implemented in accordance with the provisions of VA security system development life cycle as outlined in NIST Special Publication 800-37, Guide for Applying the Risk Management Framework to Federal Information Systems, VA Handbook 6500, Information Security Program and VA Handbook 6500.5, Incorporating Security and Privacy in System Development Lifecycle. The contractor/subcontractor is required to design, develop, or operate a System of Records Notice (SOR) on individuals to accomplish an agency function subject to the Privacy Act of 1974, (as amended), Public Law 93-579, December 31, 1974 (5 U.S.C. 552a) and applicable agency regulations. Violation of the Privacy Act may involve the imposition of criminal and civil penalties. The contractor/subcontractor agrees to: Comply with the Privacy Act of 1974 (the Act) and the agency rules and regulations issued under the Act in the design, development, or operation of any system of records on individuals to accomplish an agency function when the contract specifically identifies: (a) The Systems of Records (SOR); and (b) The design, development, or operation work that the contractor/subcontractor is to perform; Include the Privacy Act notification contained in this contract in every solicitation and resulting subcontract and in every subcontract awarded without a solicitation, when the work statement in the proposed subcontract requires the redesign, development, or operation of a SOR on individuals that is subject to the Privacy Act; and Include this Privacy Act clause, including this subparagraph (3), in all subcontracts awarded under this contract which requires the design, development, or operation of such a SOR. In the event of violations of the Act, a civil action may be brought against the agency involved when the violation concerns the design, development, or operation of a SOR on individuals to accomplish an agency function, and criminal penalties may be imposed upon the officers or employees of the agency when the violation concerns the operation of a SOR on individuals to accomplish an agency function. For purposes of the Act, when the contract is for the operation of a SOR on individuals to accomplish an agency function, the contractor/subcontractor is considered to be an employee of the agency. Operation of a System of Records means performance of any of the activities associated with maintaining the SOR, including the collection, use, maintenance, and dissemination of records. Record means any item, collection, or grouping of information about an individual that is maintained by an agency, including, but not limited to, education, financial transactions, medical history, and criminal or employment history and contains the person s name, or identifying number, symbol, or any other identifying particular assigned to the individual, such as a fingerprint or voiceprint, or a photograph. System of Records means a group of any records under the control of any agency from which information is retrieved by the name of the individual or by some identifying number, symbol, or other identifying particular assigned to the individual. 7 6 The vendor shall ensure the security of all procured or developed systems and technologies, including their subcomponents (hereinafter referred to as Systems ), throughout the life of this contract and any extension, warranty, or maintenance periods. This includes, but is not limited to workarounds, patches, hotfixes, upgrades, and any physical components (hereafter referred to as Security Fixes) which may be necessary to fix all security vulnerabilities published or known to the vendor anywhere in the Systems, including Operating Systems and firmware. The vendor shall ensure that Security Fixes shall not negatively impact the Systems. The vendor shall notify VA within 24 hours of the discovery or disclosure of successful exploits of the vulnerability which can compromise the security of the Systems (including the confidentiality or integrity of its data and operations, or the availability of the system). Such issues shall be remediated as quickly as is practical, but in no event longer than 10 days. When the Security Fixes involve installing third party patches (such as Microsoft OS patches or Adobe Acrobat), the vendor will provide written notice to the VA that the patch has been validated as not affecting the Systems within 10 working days. When the vendor is responsible for operations or maintenance of the Systems, they shall apply the Security Fixes within 10 days. All other vulnerabilities shall be remediated as specified in this paragraph in a timely manner based on risk, but within 60 days of discovery or disclosure. Exceptions to this paragraph (e.g. for the convenience of VA) shall only be granted with approval of the contracting officer and the VA Assistant Secretary for Office of Information and Technology. INFORMATION SYSTEM HOSTING, OPERATION, MAINTENANCE, OR USE For information systems that are hosted, operated, maintained, or used on behalf of VA at non-VA facilities, contractors/subcontractors are fully responsible and accountable for ensuring compliance with all HIPAA, Privacy Act, FISMA, NIST, FIPS, and VA security and privacy directives and handbooks. This includes conducting compliant risk assessments, routine vulnerability scanning, system patching and change management procedures, and the completion of an acceptable contingency plan for each system. The contractor s security control procedures must be equivalent, to those procedures used to secure VA systems. A Privacy Impact Assessment (PIA) must also be provided to the COTR and approved by VA Privacy Service prior to operational approval. All external Internet connections to VA s network involving VA information must be reviewed and approved by VA prior to implementation. 8 7 Adequate security controls for collecting, processing, transmitting, and storing of Personally Identifiable Information (PII), as determined by the VA Privacy Service, must be in place, tested, and approved by VA prior to hosting, operation, maintenance, or use of the information system, or systems by or on behalf of VA. These security controls are to be assessed and stated within the PIA and if these controls are determined not to be in place, or inadequate, a Plan of Action and Milestones (POA&M) must be submitted and approved prior to the collection of PII. Outsourcing (contractor facility, contractor equipment or contractor staff) of systems or network operations, telecommunications services, or other managed services requires certification and accreditation (authorization) (C&A) of the contractor s systems in accordance with VA Handbook 6500.3, Certification and Accreditation and/or the VA OCS Certification Program Office. Government-owned (government facility or government equipment) contractor-operated systems, third party or business partner networks require memorandums of understanding and interconnection agreements (MOU-ISA) which detail what data types are shared, who has access, and the appropriate level of security controls for all systems connected to VA networks. The contractor/subcontractor s system must adhere to all FISMA, FIPS, and NIST standards related to the annual FISMA security controls assessment and review and update the PIA. Any deficiencies noted during this assessment must be provided to the VA contracting officer and the ISO for entry into VA s POA&M management process. The contractor/subcontractor must use VA s POA&M process to document planned remedial actions to address any deficiencies in information security policies, procedures, and practices, and the completion of those activities. Security deficiencies must be corrected within the timeframes approved by the government. Contractor/subcontractor procedures are subject to periodic, unannounced assessments by VA officials, including the VA Office of Inspector General. The physical security aspects associated with contractor/subcontractor activities must also be subject to such assessments. If major changes to the system occur that may affect the privacy or security of the data or the system, the C&A of the system may need to be reviewed, retested and re-authorized per VA Handbook 6500.3. This may require reviewing and updating all of the documentation (PIA, System Security Plan, Contingency Plan). The Certification Program Office can provide guidance on whether a new C&A would be necessary. The contractor/subcontractor must conduct an annual self assessment on all systems and outsourced services as required. Both hard copy and electronic copies of the assessment must be provided to the COTR. The government reserves the right to conduct such an assessment using government personnel or another contractor/subcontractor. The contractor/subcontractor must take appropriate and timely action (this can be specified in the contract) to correct or mitigate any weaknesses discovered during such testing, generally at no additional cost. VA prohibits the installation and use of personally-owned or contractor/subcontractor-owned equipment or software on VA s network. If non-VA owned equipment must be used to fulfill the requirements of a contract, it must be stated in the service agreement, SOW or contract. All of the security controls required for government furnished equipment (GFE) must be utilized in approved other equipment (OE) and must be funded by the owner of the equipment. All remote systems must be equipped with, and use, a VA-approved antivirus (AV) software and a personal (host-based or enclave based) firewall that is configured with a VA-approved configuration. Software must be kept current, including all critical updates and patches. Owners of approved OE are responsible for providing and maintaining the anti-viral software and the firewall on the non-VA owned OE. All electronic storage media used on non-VA leased or non-VA owned IT equipment that is used to store, process, or access VA information must be handled in adherence with VA Handbook 6500.1, Electronic Media Sanitization upon: (i) completion or termination of the contract or (ii) disposal or return of the IT equipment by the contractor/subcontractor or any person acting on behalf of the contractor/subcontractor, whichever is earlier. Media (hard drives, optical disks, CDs, back-up tapes, etc.) used by the contractors/subcontractors that contain VA information must be returned to the VA for sanitization or destruction or the contractor/subcontractor must self-certify that the media has been disposed of per 6500.1 requirements. This must be completed within 30 days of termination of the contract. Bio-Medical devices and other equipment or systems containing media (hard drives, optical disks, etc.) with VA sensitive information must not be returned to the vendor at the end of lease, for trade-in, or other purposes. The options are: Vendor must accept the system without the drive; VA s initial medical device purchase includes a spare drive which must be installed in place of the original drive at time of turn-in; or VA must reimburse the company for media at a reasonable open market replacement cost at time of purchase. Due to the highly specialized and sometimes proprietary hardware and software associated with medical equipment/systems, if it is not possible for the VA to retain the hard drive, then; The equipment vendor must have an existing BAA if the device being traded in has sensitive information stored on it and hard drive(s) from the system are being returned physically intact; and Any fixed hard drive on the device must be non-destructively sanitized to the greatest extent possible without negatively impacting system operation. Selective clearing down to patient data folder level is recommended using VA approved and validated overwriting technologies/methods/tools. Applicable media sanitization specifications need to be pre-approve...