Our understanding of food security in sub-Saharan Africa (SSA) has been hampered by limitations in the temporal and spatial representativeness of data. Food balance sheets provide scalable estimates of per capita food availability, but fail to represent food access, stability and their causal linkages. In contrast, rural household surveys represent detailed conditions for one or multiple points in time, but are influenced by survey timing and are often limited in geographical coverage. This study draws on a large sample of rural land-holding households in SSA (n = 6,353) to identify household level food access deficiencies and to understand the associations with rural livelihoods and food sourcing behavior throughout the year. Food access deficiencies were identified using food security of access and diet diversity indicators. Dietary diversity and channel of access (farm or purchased) were enumerated for the “flush” and “lean” periods and food security of access was enumerated for the lean period only - making the results of this study independent of survey timing. As many as 39% of households were classified as severely food insecure (in terms of food access) and as many as 49% of households were likely to be deficient in micronutrients in the lean period. Vulnerability to food insecurity and micronutrient deficiencies differed by household composition, agricultural livelihood characteristics and agro-ecological zone. Dairy, fruit and vitamin A-rich produce were predominantly accessed through the farm channel. Households with a livestock component to their farm had a lower prevalence of severe food insecurity and higher diet diversity scores. These findings have implications for the development of nutrition-sensitive and nutrition-specific interventions. Interventions need to be tailored to agro-ecological zone, household composition, scale of operation and production mix. Increasing income will not necessarily result in improved diet diversity or healthy dietary choices. Interventions focused on income generation should monitor and promote crop and livestock production diversity and provide nutrition education.