Obstacles to extending the range and availability of providers

As a newly-developing field, LLN provision faces a 'chicken and egg' situation where it is difficult to expand the range of provision without a commensurate increase in demand and vice versa. This uncertainty of demand is probably one of the main reasons that many providers (including ITPs) employ most tutors on a part-time basis and short-term contracts.

The SWLP project reports show financial viability to be one of the main factors affecting the availability of workplace literacy providers. Anecdotal evidence about providers leaving the field tends to support these observations, and that voluntary organisations such as Literacy Aotearoa and ESOL Home Tutors face a constant challenge of losing their trained tutors to paid providers such as TEIs or workplace providers.

Some community ESOL and adult literacy programmes run waiting lists because at a particular time they don't have the right mix of tutors to match learners one-to-one, or they have insufficient resources to run small groups, but there is no systematic information about that issue.

Integrated provision

LLN-specific courses focus solely on the teaching of LLN skills. Integrated programmes include explicit LLN skills teaching as well as (and equally important to) the skills and content related to another course-such as retailing, building or food processing. Integrated provision has become a major platform of the foundation learning sector with policy and funding arrangements geared to providers integrating LLN into vocational, pre-employment and workplace programmes. This makes it challenging to differentiate LLN programmes from other low NQF programmes, which may include learners with LLN needs.

International research points to benefits from integrating LLN into other contexts provided this provision involves literacy tutors as well as vocational tutors. The most significant study on integrated learning internationally thus far involved 1,916 learners on 79 vocational programmes in the UK (Casey, 2006). The research developed a four-point scale to determine the extent of embedding (integrating) LLN into the vocational context from the learners' point of view. On a course that was not embedded, any LLN teaching that took place would be separate and distinct; on fully-embedded programmes LLN was treated as integral to the vocational programme. The results clearly showed that learners on vocational programmes where literacy was fully embedded have higher retention and success rates and achieve better in literacy, language and numeracy qualifications.

Most importantly from the New Zealand perspective, Casey's study stresses that "when a single teacher is asked to take dual responsibility for teaching vocation skills and LLN the probability of learners succeeding with literacy and numeracy qualifications is lower" (p 6). Formal shared and ongoing planning and team work between vocational and LLN specialist tutors were identified as key factors for bringing about learner gain in the UK research. This is very different from New Zealand, where a lot of emphasis appears to be going on improving the ability of vocational tutors to integrate literacy into their teaching, without necessarily a sustained partnership with a literacy tutor.

There have only been two research studies specifically into integrated practice, both done as the concept was being introduced (Skill New Zealand, 2002a; Workbase, 2001a). We believe that in many programmes where integration is supposedly taking place, there is probably very little direct literacy teaching of LLN skills (Benseman, Lander, & Sutton, 2005; Gibbs, 2005; Skill New Zealand, 2002a; Sutton, 2004). Interestingly, one study of providers mentioned concerns from literacy tutors about a lack of professional development for them when they work for generic training providers, and a lessening of importance of specialist literacy skills with the introduction of the integrated approach (Neilson & Culligan, 2005).

Research into integrated provision is important if we are to be confident that our models of integrated practice are in fact achieving the anticipated learner LLN skills gain that UK research has demonstrated to be possible.

Specialist LLN teaching expertise is required to enable the integration of LLN into vocational programmes. We need to understand what can be realistically achieved by upskilling vocational tutors, investigate the extent to which integrated programmes explicitly teach literacy, language and numeracy and understand the nature of the specialist support required.

Quality of programmes

The Foundation Learning Pool has funded the growth in LLN-related programmes. Funding is available for intensive programmes (100+ hours per year) and to enhance the LLN component of Student Component or TO- and YT-funded programmes. In 2007, approximately 100 providers were funded to run 240 programmes, worth approximately $16 million.^[11] Learner numbers for 2007 are unknown as yet, but the growth is obvious-in 2005 approximately 3,200 learners participated in 54 Pool-funded programmes worth $3 million. The only published review of the fund was just after it started (Ministry of Education, 2003b), when 48 programmes were funded from 28 providers.

Overall, there is very little research on teaching and learning strategies and approaches, other than the development work done on reading and numeracy within the Learning for Living project.

Tutors and providers would benefit from easier access to information about innovative programmes inside New Zealand-most information on programmes appears to be presented only in monitoring reports to funders. The major national regular communication mechanisms to tutors thus far have included Literacy Works, the Literacy Portal and the Learning for Living newsletters and communication workshops. More use could be made of the insights from Foundation Learning-funded programmes presented in monitoring reports. Other opportunities include directed discussion lists in the USA^[12] or a website for LLN practitioners.

Given the rapid growth of this funding, it is timely to have a comprehensive review of the Foundation Learning Pool-the types of programmes, recruitment and retention issues in those programmes, evidence of learner gain and the issues providers have found in endeavouring to grow their programmes quickly.

There would be value in new and different mechanisms for disseminating information and ideas about good practice and innovative teaching and learning and provider development.

Improved LLN skills

Many studies reported increases in LLN skills, but only a small number provided statistical evidence of learner gain-the Learning for Living projects, and single studies on ESOL and family literacy. All the statistics on learner gain in these studies were reviewed by Dr Gavin Brown.

The Learning for Living projects involve professional development for tutors to integrate specific teaching strategies^[15] related to either reading or numeracy into their practice. Both strands involved the pre- and post-testing of learners, with the hypothesis that learners would improve their achievements as a result of their teachers' participation in professional development (PD). Aggregated findings from three reading cluster reports (Chandler, 2006; Doyle, 2006; Young, 2006) showed that 70 percent of the 149 learners with two data points made 'modest gain'.^[16] The 149 learners were approximately 60 percent of the original cohort of 252 learners. Learners with higher ratings in the pre-test improved more, which is understandable given the short time between assessments. Feedback from learners suggested they had become more confiden in test-taking and could articulate more reading strategies.

The statistical review by Brown notes that Pasifika students, students from non-English speaking homes and female students from Auckland appeared to have made very poor gains in reading in these projects. He questions whether the degree of gain is sufficient or different from that achieved by any other teaching programme.

The equivalent reports on the three numeracy clusters (Higgins, 2006; Meaney, 2006; Thomas, 2006) reported data on 194 learners (74 percent of the original cohort of 261). The proportion of competent learners increased on average 19 percent across the seven numeracy domains of the Number Framework that was used to measure progress. Learners had not improved in confidence, but felt better about how they were being taught by the second assessment. Gain appeared to be greater with women, those receiving 40+ hours teaching and those aged over 25. However, 50 percent of learners in the numeracy clusters appear not to have made measurable improvements.

The statistical review said a clear pattern of increased scores for some learners was evident but that it was not possible to ascertain the statistical significance of these results. Brown argued that it is not clear whether the gains exceeded chance or whether the teaching program had caused the gain.

One of the small number of studies focusing on ESOL endeavoured to determine the literacy gain made by low-level or pre-literacy ESOL learners on a one-semester literacy course (Shameem, McDermott, Blaker, & Carryer, 2002). In total, 118 learners took part, but only 62 had data from pre- and post-testing (approximately 50 percent of the initial cohort). Measurement of progress was by both self-report and performance assessment. There was statistically significant improvement on the performance assessments for those on the 12-hour per week writing programmes (240 hours in total) and for those who had a bi-lingual tutor for reading. The statistical review highlighted the small sample and large numbers of non-completers, as well as possible alternative explanations for the large learner gain.

The summative evaluation of the Manukau family literacy programmes (MFLP) (Benseman & Sutton, 2005) reported on 70 adults and 70 children. In the MFLP model, the adult learners attend a full-time programme tertiary education programme on-site in a school and take part in literacy-related activities with their children and in parent education. Pre- and post-data was available on 23 adults (based on self-reporting of LLN and confidence) and 21 children (based on Running Records). The adults reported large improvements. The statistical review highlighted that the children's gains were not demonstrably different from a control group of children and cautioned about the 'halo effect' that results from self-report.

Measuring gain is complex, because learners make variable gains for a variety of reasons including their skills at the commencement of the programme, the fit of the programme with their personal circumstances and their readiness to learn. Ethnicity should be considered as a variable when considering learner gain. It is not possible to compare gains across programmes because they do not share a common assessment tool, so the degree of impact on learners' LLN skills remains a largely uncharted territory.

The review of the learning gain statistics highlighted a number of issues that researchers need to more carefully consider, including the type of instruments used to measure gain, whether the teaching intervention can reasonably be considered the cause of any improvement and the quality of reporting. Appendix A discusses these issues in more depth.

Clearer data on learner gains in LLN will come when there is a national assessment tool that shows progress on the Foundation Learning Progressions to give us both specific and nationally standardised data. The tool needs to be designed and tested with practitioners and a wide range of learners across different programmes and in a variety of contexts. Extensive professional development on assessment will need to accompany the tool.

Gains in confidence

Like many social intervention programmes, the most commonly reported outcome from LLN programmes is an increase in personal confidence (Benseman, 2004a; Benseman & Tobias, 2003; Gibbs, 2005; Kempton, 2005; May, Hill, & Donaghy, 2004; Skill New Zealand, 2002b), although the studies used different terminologies and approaches to measure the gain.

Often dismissed as a 'soft outcome' that is difficult to measure and deemed to be of lesser value than improvements in LLN skills, self confidence is seen by learners and tutors as a valuable outcome and is increasingly being recognised internationally (Eldred, 2002). In reporting the high number of respondents who had been out of LLN programmes for more than two years, Benseman and Tobias (2003, p. 23) described the significance of improving self-confidence for LLN learners.

The relationship between gains in self-confidence and gains in literacy skills is clearly complex, but they are strongly inter-related. At least two forms of literacy-related confidence were referred to [in their interviews]. Some said that their increased ability to read, spell, write, solve maths problems, or speak in a group (including in English as a second language) had led to increased confidence in these skills, which in turn had led to greater levels of general self-confidence.

There were others who claimed that they had made little or no progress in improving these skills. On the other hand, while some of these interviewees said that there had been little, if any, change in their levels of confidence (with one or two stating that the programme had an adverse effect), others said that their self-confidence had increased nonetheless, as they had discovered that they were not alone with their difficulties and they had developed further strategies and ways of dealing with their difficulties. One of the most common comments from these people was that they said they had gained the confidence to tell people of their difficulties and ask for help.

In other words, improvements in literacy skills almost invariably resulted in greater self-confidence, but improvements in self-confidence (through learning self-assertions skills for example) may occur with little or no change in literacy skills. In the latter case however, improved levels of self-confidence can still give the participants greater self-awareness and a wider range of skills to manage their literacy difficulties in their daily lives.

Increased confidence leads to increased or different LLN practices-more reading on the job or for pleasure, taking on writing tasks not previously attempted, greater willingness to answer questions, helping children with their homework, more reading at work (Benseman, 2004a; Benseman & Tobias, 2003; May, Hill, & Donaghy, 2004; McDermott, 2004; Roorda, van Wichen Miller, Watson, & Poppelwell, 2005). Again, the diversity in reporting and the lack of a framework means data can't be meaningfully aggregated.

Developing a common measure of changes in confidence and literacy behaviours that providers could use alongside other outcome measures would provide consistent and quantifiable results in an area that teachers and learners think as important as LLN skills gain.

Course completion

Course completion is often used as a proxy measure of literacy skills, but the problem of attributing LLN gains comes when courses do not include any direct measures of literacy. Course completion of L1-3 certificates nationally is low (Scott, 2004). Some studies reviewed reported generally on course completion or unit standard achievements (Benseman, 2004a; Castalia Strategic Advisors, 2005; Ministry of Education, 2006b; Skill New Zealand, 2002b; Sutton, Lander, & Benseman, 2005), but the findings are limited.

Professional development activities

Achieving learner gain is, of course, linked to the quality of the teaching offered in LLN provision and developing qualifications has been recognised as a key component of raising quality.

Before 2006, there were no specialist LLN qualifications available for LLN teachers in New Zealand. Now there are two National Certificates, one for specialist LLN teachers and the other for vocational tutors or workplace trainers who integrate literacy into other programmes-both are at NQF Level 5. The Ministry of Education has provided up to 500 scholarships for students as an incentive to enrol in these programmes. In 2007 AUT introduced a Master in Adult Literacy and Numeracy Education, but there is still a gap at Diploma and Bachelor levels. There are a number of providers offering these programmes, but it is unclear yet whether the demand is sufficient to keep all courses viable.

If LLN provision is to be expanded the LLN tutor workforce needs to grow. Research about the numbers and characteristics of LLN specialist tutors, their skills and motivations, sources of professional development, and conditions of employment would help workforce planning and successful recruitment.

An evaluation of the impact of the new qualifications is also important for long term workplace planning. This evaluation would need to investigate what sorts of tutors are being recruited into study and document changes in teaching practices and whether qualifications impact on job retention and changes in employment conditions.

It is particularly important to understand the impact on the workplace trainers and vocational tutors who are doing the National Certificate in Adult Education and Training (Foundation Educators) to provide greater insight into integrated programmes.

The Learning for Living professional development project has been the largest PD initiative in the sector to date. The focus on professional development related to LLN subject knowledge and teaching strategies in reading and numeracy was informed by the literature review on effective LLN teaching (Benseman, Sutton, & Lander, 2005) and the findings from the first round of Learning for Living exploratory projects.

Five or six providers were recruited in geographic clusters. Two tutors attended regular PD sessions with their manager and the tutors practised using new approaches in their classes between PD sessions. Reading clusters ran in Waikato/Bay of Plenty, Wellington and Auckland, and numeracy clusters ran in Christchurch, Auckland and Central North Island. External developers ran the cluster meetings and acted as coaches for each provider. An evaluation project run by external evaluators has run alongside the projects. The latter stages of the project have involved managers and tutors endeavouring to roll out these new approaches more widely across their organisations. A series of communications workshops is currently being held throughout the country to disseminate the findings from this project (Jakob-Hoff & Postlethwaite, 2006), and a written handbook for tutors on the new approaches is being produced to supplement these workshops.

Tutors in both the reading and numeracy strands have reported increased knowledge about those subjects, and appreciated the sustained opportunity for professional development. In the numeracy strand, the least experienced tutors reported gaining the most. Providers have responded positively to the programme as well. The data on tutors' changes in practice is based on self-report.

Given the considerable investment in the Learning for Living project and the time it takes to really bed in changes in practice, a follow-up is needed of the participating organisations and tutors to determine the degree to which the changes are implemented in daily teaching over the longer term. Identifying how the lessons from the Learning for Living project will be rolled out through the sector as a whole is also important.

The observation study of 15 LLN teachers (Benseman, Lander, & Sutton, 2005) also pointed to issues about broad teaching skills, not just LLN-specific ones. Some teachers lacked questioning skills and used a limited range of teaching strategies. The findings are illustrative only because the sample was small, but they point to the need for professional development on ways to increase learner involvement in lessons and to give learners more focused feedback. There is increasing emphasis internationally on the role of formative assessment in improving the quality of teaching and related learner gain in adult LLN, including skills such as questioning and giving feedback to learners. Formative assessment focuses specifically on the interactions between teachers and learners and has the potential to enhance both general teaching and subject specific skills in LLN. Formative assessment has had a major impact on learner achievement in schools, particularly for low performing students. This led to an OECD study^[17] on the use of formative assessment with adult foundation learning which will finish early 2008; the Ministry of Education commissioned a background report on formative assessment in New Zealand during 2006 as part of that study. Formative assessment has the potential to play a major role in the development of quality teaching and needs to be considered in the mix of professional development activities being planned by the TEC. PD activities in formative assessment are probably best placed to happen within individual organisations.

Professional development on assessment, in particular on formative assessment, would complement the professional development activities thus far. Individual organisations would benefit from in-house formative assessment projects.

Some tutors may not have the literacy skills necessary to participate in a PD programme or to teach reading and writing confidently (Jakob-Hoff, Postlethwaite, Stokes, & Talagi, 2003; Sutton, 2004).^[18] Lack of LLN teaching skills was also a finding from this project which involved 80 tutors.

There have been other significant PD activities-for example, Workbase's programme of intensive reading and writing workshops and the programmes for member schemes run by Literacy Aotearoa and ESOL Home tutors, but these have not been formally evaluated.

Assessment tools

The assessment methodologies and processes used in foundation learning are varied and not always rigorous. Often more emphasis goes on initial than summative assessment (Hattie, Sutton, & MacKay, 2007; NZCER, 2006). Many programmes use Individual Learning Plans and the identification and achievement of learners' goals as the assessment drivers and the principal measurement method is learner self-report. The standardised assessments that are used tend to be short tests that lead to scores, which are of limited diagnostic value (Hattie, Sutton, & MacKay, 2007).^[19] Tutors spend time and energy making assessment resources from scratch and contextualising them to each situation or course because contextualised teaching, learning and assessment is thought to be more effective than standardised approaches (NZCER, 2006; Sutton, 2004).

Many studies identified the need for a system that provides reliable and standardised national outcome data (Benseman, Sutton, & Lander, 2005; Hattie, Sutton, & MacKay, 2007; Kempton, 2005; NZCER, 2005; Ramasamy, Pusch, & Clark, 2005; Roorda, van Wichen Miller, Watson, & Poppelwell, 2005; Sutton, 2004). The May 2007 budget allocation for a national adaptive computer-based foundation assessment tool for foundation learning provides an opportunity to achieve this. The ability to do fast initial screening of general groups of people (in particular, beneficiaries), to identify whether a more in-depth assessment of their LLN skills is warranted, has also been identified as an issue (Ramasamy, Pusch, & Clark, 2005; Sutton & Benseman, 2005) and a project is underway to adapt a UK screening tool for use here.

Generally tutors are enthusiastic about taking part in training on assessment, and many are coming to see value in standardised assessment. This was borne out in the Learning for Living clusters, where tutors found the diagnostic interview on numeracy and the focus groups used to assess learners' understandings of reading to be particularly insightful.

Foundation learning progressions

The draft Foundation Learning Progressions^[20] released by the TEC in 2006 endeavour to identify the common sequence of knowledge and skills that adults need to develop in order to reach foundation level competence (and be prepared for lifelong learning). The progressions cover seven strands (speaking, listening, reading, writing, number, reason statistically and measurement) over six steps. Step 1 is entry level and Step 6 broadly equates to NCEA Level 2 (Year 12) or Level 2 on the NQF. The progressions reflect 'patterns of progress' rather than a prescribed linear sequence of learning.

The Progressions are designed to provide a common language and framework describing the learning that is needed to 'equip adults to be independent citizens, workers, family and community members.' Consultation for the draft progressions ended June 2007.

The New Zealand Literacy Portal

The New Zealand Literacy Portal (http://www.nzliteracyportal.org.nz/) is a free online information database funded by the TEC and run by Workbase. It is designed to provide a knowledge base of adult literacy information contributed by both New Zealand and international organisations. The Portal links to over 1,700 free items on topics including e-literacy, ESOL, financial literacy, health literacy, learning disabilities, literacy, numeracy and te reo Maori. Content includes:

• practical resources for use with learners
• resources for staff professional development
• research reports and case studies
• journal articles and newsletters
• a calendar of events
• links to adult literacy discussion lists.

The service was set up in 2003 and receives about 2200 visits (of 10+ minutes) per month. To date no formal evaluation has been carried out on the nature of the clientele, its patterns of usage or ratings of its utility and value, although several small in-house reviews have been carried out by Workbase.